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ABSTRACT 

This guide is intended to assist states and service 
delivery areas (SDAs) in addressing the new oversight 
responsibilities and opportunities stipulated by the Job Training 
Partnership Act (JTPA) with respect to net impact evaluations. The 
first chapter contrasts the objectives and provisions of JTFA with 
those of the Comprehensive Employment and Training Act (CETA). The 
literature review provided in the second chapter discusses previous 
studies that have analyzed the impacts of various employment and 
training programs and examines the implications of previous research 
for developing a state-level net impact model. The third chapter 
presents a conceptual framework for developing a state-level JTPA net 
impact mode. The conceptual framework, which builds on the results of 
the literature review, indicates the key outcome measures that should 
be examined, the subgroups of trainees for which impacts should be 
measured separately, the program activities (treatments) that should 
be examined, the types of economic and demographic characteristics 
that may affect outcomes, and the data sources that will bejised to 
measure the aforementioned elements. A chapter on research design 
discusses various issues relating to the participant sample, use of a 
comparison group strategy, and sample sizes for the participant and 
comparison samples. The final chapter outlines a data analysis plan 
that addresses the issues of examining the adequacy of the 
comparision groups, estimating the net impacts of JTPA programs, 
obtaining net impact estimates for various subgroups, dealing with 
measurement error and other statistical issues, and making 
adjustments for potential data and design deficiencies. A sample JTPA 
application form is appended. (MN) 
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CONTEXT OF THIS VOLUME 

This is one in a series of volumes produced by the JTPA EVALUATION DESIGN PROJECT. 



PURPOSE AND PHILOSOPHY 

The purpose of this project has been to develop a set of evaluation tools that are useful to states and local service delivery 
areas (SDAs) in judging the way their JTPA programs are being managed and the impact they are having. The intention 
•if. J rffu anal ^ 1C an . d mana .8 er ' al tools on sound program concepts and research methods, and to design 
«?»/?. ^ i«S . .' t?™.?- °," ?- ta - ined 1S - of . pra , ctlcal and dirsct use in '""Proving JTPA policies and programs at the 
md^daSo^mS JTPA 0 informatl0n 15 also expected t0 make a uni 9 ue contribution to national training polic> 

It is hoped that these volumes will stimulate and support state and local evaluation efforts in JTPA, and promote more 
consistency than in previous programs with respect to the issues studied and the methods used to investigate them An 
ffiSw ! encourage the generation of complementary information on program implementation and impact 
that is comparable across states and SDAs. Comprehensive, comparable information is essential to the development of 
a valid and reliable knowledge base for resolving problems and improving programs. It is also required for adjusting na- 
tional training strategies to changing needs and priorities at the state and local level. 

PRODUCTS 

Consistent with this purpose and philosophy, the project has produced a set of materials to assist states and SDAs in 
evaluating their programs These are to be useful in placing, designing and implementing evaluation activities. As an 
integrated collection, each set is developed to support comprehensive evaluations over the JTPA planning cycle. 

The careful tailoring of these materials to state and local users is appropriate. JTPA represents a new employmeni and 
training policy shaped not only by the experience of manager; *nd the perspectives of employers, but by scientific assessments 
ot previous approaches for addressing unemployment, poverty and other barriers to economic security. In this context, 

LnH^«wfl J a EV r0 S ainS V s als0 exp , ected 10 be j ud 8 ed - In fact . the Act's assessment requirements are more explicit 
and sophisticated than those of any employment and training legislation to date. It clearly distinguishes between monitor- 
ing activities, whose purpose is to determine compliance (such as with performance standards) and evaluation activities, 
whose purpose is to determine how a program is being managed and implemented, and the kinds of effects it is having 
on recipients and relevant others. Equally significant, new constitutencies are expected to make these more rigorous 
assessments. States and SDAs now have this important responsibility. It is the first time in the history of employment 
and training programs that the Federal government's evaluation role has been significantly reduced. 

rt J, h J S ^ han8e af ^ r £. stat " and locaI area . s opportunities to influence public policy. It also requires them to assume new 
oversight responsibilities. Program evaluation is expected to become an integral part of the management of organizations 
administering, planning and delivering public training services. This is as it should be. The more information available 
at these levels, where changes in organizations can most readily be made, the more effective the management of JTPA 
programs. This project was undertaken in that context. 

The evaluation tools produced by the project have been developed with a sensitivity to the differing needs, interests 
n r i S ™ S i° S , tateand loc L al ^ seTS - They have been P= r kaged into a single comprehensive and integrated set of volumes 

called JTPA Evaluation at the State and Local Level. The set contains planning and evaluation guides and issue papers. 

The following volumes are available in the set: *^ 
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Volume 



Author 



I: Overview 

II: A General Planning Guide 
III: A Guide for Process Evaluations 
III Supplement: Some Process Issues at the State Level 
IV: A Guide for Gross Impact Evaluations 
V: A Guide for Net Impact Evaluations 
VI: An Implementation Manual for Net Impact Evaluations 
VII: Issues Related to Net Impact Evaluations 

A. Issues in Evaluating Costs and Benefits 

B. The Debate Over Experimental vs. Quasi-Experimental Approaches 
VIII: MIS Issues in Evaluating JTPA 

^2V:\d hi ' : "? h e f ch °/ ihQ di ? crete P r0 . ducts listed above is the responsibility of a single author, each seeks to incor- 
porate the results of professional peer review, the many excellent recommendations of the advisory group, and the ideas 
and suggestions of the numerous practitioners interviewed in the process ot developing these materials 



Project Team 
Deborah Feldman 
David Grembowski 
David Grembowski 
Carl Simpson 
Terry Johnson 
Terry Johnson 

Ernst Stromsdorfer 
Ann Blalock 
David Grembowski 



To further qualify these volumes, Volume HI is accompanied by a supplement for state users. This is consistent with 
ShS!fS W^?*™" . stat « a P d SDAs in the kinds of process issues that are most £££ES. The 
volume on net impact evaluations is sufficiently technical, because of the statistical methods involved, that a practical 
manual has been written to accompany it. This guide and manual tend to be more appropriate for states, since relatively 

Sff."s3S?£? "Vf^ , f0r ^ aly u SiS - H - 0Wever ' they m e< 5 ua "y useful 10 larger SDAs and consortia of sma er 
. • y W I nt 10 JOintly study l , he net ,m P act of tneir Programs. Regional evaluations, for example, can be very 
productive in providing management information relevant to regional labor markets. Although there is a separate issue 

SfSiS wftf! ~ ^ th !^ Sue 15 51,50 COv " ed in ' h . e « ross im P act and net ™pa« guides. In this respect, 
he user benefits from three related but different approaches to this important element of program evaluations. Also, 
the user should be aware that the Appendix of Volume II includes A Report on a National/State Survey of Local JTPA 
Constituencies. This survey was carried out by Bonnie Snedeker, with the assistance of Brian O'Sullivani to provide addi- 
tional input from practitioners to the development of the planning and process evaluation guides. 
In conclusion, several expectations have directed the development of these volumes: 



THE GUIDES 

The General Planning Guide 

This guide is to assist users in planning, funding and developing an organizational capacity to carry out process, cross 
outcome, and net impact evaluations and to utilize their results. Separate state and local versions are available. 

The Evaluation Guides 
These volumes are to have the following characteristics: 

□The guides are to complement one another. 

•They are to provide information on program management and other characteristics of program implementation, which 
can : 

—Describe the way in which administrative, managerial and service delivery policies and practices operate to affect 

outcomes, as a set of interventions separate from the program's services. 
-Pinpoint the source, nature and extent of errors and biases for which adjustments must be made in gross and net 

impact evaluations. 6 
—Help explain the results of gross and net impact evaluations. 

•They are to provide information on aggregate gross outcomes, and outcomes differentiated by type of service and 
type of recipient, which can: J v 

—Describe relationships between certain implementation modes and service strategies, and a broad array of client and 
employer outcomes. 7 d,lu 

—Help explain the results of net impact evaluations. 

—Suggest the more important outcomes that should be studied in net impact evaluations. 

—Help sort out those aspects of implementation that may be most critical to study in process evaluations. 

•They are to provide information on net impact (the program's return on investment), which can: 
—Closely estimate the effect of the program's services on clients. 

—Suggest which services and client groups are most important to study in broader but less rigorous gross impact studies 
—Help identify the decision points in program implementation (particularly service delivery) whicn may be most 
important to study in process evaluations. 

□The guides are to enable the user to carry out comprehensive assessments of JTPA programs. 

•They are to allow the user to acquire several different perspectives on the same program within a particular time period: 
on program implementation, on outcomes for clients and employers and on net impact. 

•They are to permit the user to interrelate these different kinds of information to gain a wider understanding of what 
is happening in a program and why. 

□The guides are to describe approaches and methodologies as consistently as possible, to achieve comparability. 
•They are to define variables and relationships as similarly as possible. 

•They are to define research designs, and methods of data collection and analysis using as similar concepts as possible. 

□The guides are to draw from past research on employment and training programs, as well as seek new approaches and 
methods of specific value in evaluating JTPA at the state and local level. 

"evalS l ° thC CXtent V 0 ™* 1 * and feasible « the issues measures reflected in Federal monitoring and 

#Th 7xn! 10 "l akc - sel ? tiye "? sc °. f the results of relcvant CETA studics ' national studies of JTPA, and issue papers 
on JTPA evaluation by national public interest organizations in the employment and training area. 

•They are to rely on the professional literature in applied social research. 



THE ISSUE PAPERS 

t; rt y 0 T^V V1 ! COntainS tW0 is u sue ?? p . ers wh . ich J se . rve * s c ompanion pieces to the preceding volumes on net impact evalua- 
J fS i P3per °" cost - benefu ' ss L ues J? designed to help users identify, measure and analyze relationships between 
moneta y and nonmonetary costs and benefits in determining the program's return on investment. The second paper ex- 
amines the pros and cons of different research strategies associated with the net impact approach. The final volume on 
MIS issues is to assist users in better understanding how JTPA and other employment and training management informa- 
tion systems can efficiently support the evaluation of program implementation and impact. 

THE SET OF VOLUMES 

. I™ is / 73?e f rared ' but fffords flexible use. The user can utilize the entire set for comprehensive evaluations over 

IZa E >X m , g Cy • ° r l0n P r P lanni ? g P? n0d ' or the user can ap P'>' the information in each volume independently, 
based on the most pressing evaluation priorities and timeframes and given the extent of resources, during a particulv 



It should be understood that although evaluation products have been developed for JTPA, their basic principles and 
^aiSpnl^m 6 SlaleS '° Cal are3S t0 CValUate ° ther employment and trainin € Programs and 

GENERAL ACKNOWLEDGMENTS 

i^mJ^I^} 0 ^ ? ESI c N ™ 0JECT deveI °P ed and carried out based on the partnership philosophy 
; erlesthe j T u PA legislation. Several partnerships should be recognized for their substantial contributions to the 
products previewed here: the project development and coordination partnership; the public-private funding partnership; 
the interdisciplinary design partnership; and the advisory partnership. parmcrsmp, 
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CHAPTER 1. INTRODUCTION 



The Job Training Partnership Act (JTPA) of 1982, which officially 
replaced the Comprehensive Employment and Training Act (CETA) on 
October 1, 1983, continues CETA's stated objective of providing job 
training to Improve the economic well-being of trainees. In 
particular, Section 106 of the Act explicitly states that "job training 
1s an Investment 1n human capital," and that the "return on the 
Investment Is to be measured by the Increased employment and earnings 
of participants and the reduction In welfare dependency." Although the 
major objectives of the two programs are similar, the passage of JTPA 
significantly changed the administration and operation of employment 
and training programs, as well as the types of services provided. 

The major characteristics of JTPA that distinguish It from CETA and 
earlier employment and training systems Include: (1) the expanded role 
of the private sector as a partner In the planning and overseeing of 
job training services; (2) the transfer of responsibility for program 
administration, operation, and accountability from the federal 
government to the state and local level; (3) requirements on how funds 
are expended that result In programs that emphasize skills training 
rather than work experience or public service employment; and (4) a 
central role for performance standards to ensure that the employment 
and training system meets Its objectives. As part of the performance 
standards application process, the authorizing legislation requires 
governors to use six percent of the funds allocated to each state to 
either provide Incentive grants for service delivery areas (SDAs) that 
exceed their performance standards or technical assistance to SDAs that 
do not meet their standards. 

As a result of these numerous changes, states and local areas have been 
given much greater responsibility for program accountability under JTPA 
than In the past. This has, In turn. Increased the need for reliable 
analysis of program effects and outcomes at the state and local level. 
For the most part, the only reliable analyses of employment and 
training programs have been conducted at the national level. Although 
these national studies have provided much useful Information on the 
effectiveness of employment and training programs overall, they are of 
limited use to state administrators and local program operators for 
several reasons. First, the results of these studies have not been 
available on a timely basis. For example, several recently completed 
studies of the Impact of CETA programs examined the early post-program 
experiences of 1976 and 1977 enrollees. Because of the many changes In 
CETA since that time, and the major differences between CETA and JTPA, 
the usefulness of these results for guiding managerial decisions Is 
1 Imlted. 

Second, because the studies have been conducted using nationally 
representative samples of program participants, the sample sizes 
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available at the state and local level are too small to support 
reliable program analysis at a disaggregated lavel. Moreover, because 
of confidentiality concerns, local Identifiers have not been 
available. This data constraint has prevented anaiysts from both 
controlling for differences 1n program environments as well as from 
examining the extent to which program Impacts vary by labor market or 
other characteristics of the environments In which the programs operate. 

Finally, most of these studies have only examined earnings gains, 
whereas the authorizing legislation also requires an examination of 
employment gains and reductions In welfare dependency. As a result of 
these limitations, previous national studies are of little use In 
assisting states and local areas to make Informed and objective 
Judgments of the effectiveness of JTPA programs to meet their Increased 
oversight responsibilities under the Act. Moreover, while the need for 
reliable program analysis at the state and local level Is Increasing, 
the federal government Is greatly reducing Its role In providing 
evaluation research Information. 

In response to the lack of useful program evaluation Information at the 
State and local level, the National Commission for Employment Policy 
awarded a grant to the Washington State Employment Security Department 
to develop a set of Integrated models that could be used by states and 
SDAs In evaluating their JTPA programs. These models are presented In 
a set of complementary evaluation guides which are outlined In the 
Introductory context section of this volume. Included In this set 
are: (1) this volume on state-level net Impact evaluation . (2) guides 
for state and local process evaluations , and (3) a guide for state and 
local gross Impact evaluations . Since gross Impact and process 
evaluations can supplement and Inform che net Impact analysis In 
Important ways, this guide will periodically refer to these additional 
evaluation approaches. 

In this guide we outline the Issues Involved In designing a reliable 
net Impact evaluation of JTPA and present a specific evaluation 
approach or "model" for carrying out such an evaluation. More specific 
details on net Impact data collection and data analysis are contained 
In a companion volume, An Implementation Manual for Net Impact 
Evaluations (Volume 6). While this guide and the manual accompanying 
It assume a state-level analysis, larger SDAs or a consortium of SDAs 
may also readily adapt this net Impact design to meet more local-level 
analysis needs. 
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Before describing the net Impact evaluation model, It Is Important to 
understand the significance of the term "net," and how a "net" Impact 
model differs from a "gross" Impact model. A gross Impact analysis 
essentially compares the post-program labor market experiences of 
participants with their pre-program experiences and attributes all 
gains to the program. Although a carefully designed gross Impact 
analysis can provide useful Information on the relative effectiveness 
of different JTPA program activities, because numerous other factors 
may have changed from the pre-program to the post-program period that 
affect participants' labor market experiences (e.g., Improvement In 
labor market conditions), such an analysis Is not likely to Isolate the 
true benefits of program participation per se and should not be used to 
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measure tha return on the job training investment. On the other hand, 
n,r*f;*J? **!l na A ySlS com P ares the lfl bor market experiences of 
J?Uiir P n ? * * exper1ences of 3 comparison group of otherwise 

VZl Vh no "P art1ci P, a " ts - The comparison group is used to approximate 
what the Tabor market experiences of participants would have been in 
the post-program period had they not participated In the program. As 
such a net Impact analysis only attributes to program participation 
the Incremental gain In labor market experiences that occurs over and 
above what would have happened had these Individuals not participated 
in the program This is the appropriate concept for providing 
rundTJnSer°JTPA r6tUrn °" ^ 1nVestment oF J° b tra1n ^ programs 

In developing a state-level net impact model of JTPA, we were guided by 
several considerations. First, In order for the model to assist states 
tn meeting their new accountability responsibilities, the model must be 
£sabi|. In particular, It must provide meaningful Information that can 
De understood by a nontechnical audience In a cost-effective and timely 
manner. Second, to maximize usability, the model must recognize the 
severe resource constraints that states and SDAs face, as well as 
certain other practical considerations. The two most Important 
practical considerations that affected the recommended approach are (1) 
states and local SDAs will not generally be willing to implement an 
experimental design in which eligible applicants are randomly assigned 
to treatment-control status, and (2) states and local SDAs will not 
generally be willing to conduct follow-up interviews with a large 
sample of participants and comparison group members. Finally, the 
model must be designed to produce valid estimates of the net impacts of 
JTPA programs on the post-program outcomes of participants. 1 

In developing a state-level net impact model we have attempted to meet 
these objectives while recognizing the inherent tension between them, 
lo the extent that we are successful in designing a model that Is 
usable and provide-? valid results, then an Important by-product, namely 
consistency in application across states, will occur. This will 
maximize the Information obtained from the evaluations and make an 
important contribution to what is known about the effectiveness of 
employment and training programs in states that face different 
environments. 

The remainder of this report is organized as follows. In Chapter 2 we 
present a review of the previous studies that have analyzed the impacts 
of various employment and training programs. This review documents 
what is known about program impacts by indicating the outcomes 
examined, the comparison groups used, the different types of data 

1 J, hree J types of val1d1tv can be distinguished: internal validity is 
achieved when the models' estimates are unbiased estimates of the net 
impacts of JTPA; external validity Is achieved when the results can be 
generalized to the state as a whole; statistical validity Is achieved 
when the power of the analysis Is sufficient to produce statistically 
reliable estimates of the impact of JTPA. Our approach to designing a 
net Impact model to achieve these three types of validity Is described 
in Chapters 4 and 5. 
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sources used, the subgroups for which separate Impacts were estimated, 
the statistical techniques used, and the net Impact results obtained. 
The Implications of previous research for developing a state-level net 
Impact model are also discussed. 

In Chapter 3 we present a conceptual framework for developing a 
state-levei JTPA net Impact model. The conceptual framework builds on 
the results of the literature review and Indicates the key outcome 
measures that should be examined, the subgroups of trainees for which 
Impacts should be separately measured, the program activities 
(treatments) that should be examined, the types of economic and 
demographic characteristics that may affect these outcomes, and the 
data sources that will be used to measure these elements. 

In Chapter 4 we discuss several Issues Involved In developing a 
research design for analyzing the net Impacts of JTPA and present our 
recommended approach. The most difficult research design task Is the 
development of a reliable comparison group. The comparison group of 
nonpartldpants must be similar to participants on characteristics that 
affect program outcomes, and comparable data must be available for both 
groups. In this chapter we discuss alternative comparison group 
strategies and present our recommended approach. We also describe a 
plan for selecting a representative sample of JTPA participants and 
sufficient numbers of participants and comparison group members to 
provide valid results. 

In Chapter 5 we describe a general approach to analyzing the data to 
estimate the net Impacts of JTPA programs on participants' post-program 
outcomes. This chapter describes methods for examining the adequacy of 
the comparison groups selected, as well as procedures for estimating 
the net Impacts of JTPA on the key participant post-program outcomes 
Indicated In the legislation. We also discuss potential threats to the 
validity of the analysis and techniques for adjusting for such problems. 
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Evaluations off Pre-CETA and Other Employment and Training 
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Evaluations of CETA Programs 

Recent Methodological Developments 
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CHAPTER 2. LITERATURE REVIEW 



The literature analyzing the Impact of various employment and training 
programs has grown considerably over the past ten years, with the 
exception of several recent impact evaluations of CETA, most of these 
studies examined the Impact of CETA's predecessors, f or example, HDTA 
Institutional and on-the-job training programs. Some of the 
evaluations dealt with CETA contemporaries, such as the WIN program, 
the Job Corps, or special employment and training demonstrations. Even 
though none of these studies dealt explicitly with JTPA programs, and 
many focused on pre-CETA programs, they are of interest, both because 
the programs that were examined have many characteristics In common 
with JTPA programs and because the evaluation issues addressed are In 
many cases Identical to those that must be faced in developing a JTPA 
state-level net impact model. However, the differences between JTPA 
and CETA, and the even greater differences between JTPA and pre-CETA 
programs and other employment and training programs, must be kept In 
mind when determining the Implications of these studies for analyzing 



In this chapter we review the most relevant previous studies of the 
impact of employment and training programs on participants' 
post-program outcomes. Our review summarizes what 1s know about 
program Impacts by documenting the outcomes examined, the comparison 
groups used, the different types of data sources used, the subgroups 
for which separate Impacts were estimated, the statistical techniques 
used, and the net Impact results obtained. It should be noted that 
because virtually all of these studies relied on large-scale data bases 
on program participants that were created well before the analysis was 
undertaken, little (if any) Information was generally available on the 
specific content of the services provided, whether the program services 
were provided as planned, and the extent to which the services provided 
varied across sites or over time. The analysis of such Information 
(which 1s the subject of Volume III 1n this series, A 6u1de for Process 
Evaluations) would have been particularly useful 1n understanding the 
large variation 1n net Impact estimates described In this chapter. 

At the same time that the program evaluation literature has been 
growing, a complementary theoretical literature focusing on 
methodological techniques that can be used to adjust for various 
potential selection biases in measuring program Impacts has 
developed.' In this chapter we also briefly summarize the 
implications of the most relevant methodological studies for conducting 



The exact nature of selection bias, what 1t 1s, what Its sources 
are, and what can be done to account for Its presence, 1s discussed In 
Chapters 4 and 5. 
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a state-level net Impact analysis of JTPA. 



We begin with a d1sr:uss1on of evaluations of pre-CETA programs and of 
other non-CETA programs. We then review the recent studies of the net 
Impact of CETA programs on participants' post-program labor market 
experiences and briefly summarize a few recent contributions to the 
methodological literature. We conclude this chapter with a discussion 
of the Implications of these studies for conducting a net Impact 
analysis of JTPA at the state level. 



EVALUATIONS OF PRE-CETA AND OTHER EMPLOYMENT AND TRAINING 
PROGRAMS 



Two Important contributions to measuring the Impact of training 
programs on participants' post-program earnings are those by 
Ashenfelter (1975, 1978). Ashenf elter ' s approach builds on the 
availability of longitudinal Social Security Administration (SSA) 
earnings records to measure the net effect of training programs on 
earnings. He uses data on HDTA classroom training participants who 
entered training In the first three months of 1964 and a sample of 
nonpartlclpants drawn from the Continuous Work History Sample (CWHS). 
Because the participant and comparison groups are drawn from very 
different populations, Ashenfelter attempts to control statistically 
for differences between the two groups by specifying an earnings 
function that would prevail for both groups In the absence of 
training. In his model, current earnings are assumed to be the sum of 
an autoregresslon In earnings (I.e., several years of pre-program 
earnings are Included as control variables) plus a 
components-of-varlance error term to control for personal and time 
effects. 3 Assuming that the earnings generating functions are of the 
same form for participants and nonpartlclpants, Ashenfelter presents 
numerous estimates of net program Impacts. 

Ashenfelter' s model produces estimates of the net program Impact In 
each of the five years following training taken In 1964, disaggregated 
by race and sex groups. His results Indicate that training Increases 
the earnings of all subgroups. For men, the effect Is between $150 and 
$500 per year Immediately following training, but declining to 
approximately half this figure five years later. For women, the effect 
ranges between $300 and $600 Immediately following training and does 
not appear to decline In subsequent years. 

Although Ashenfelter's empirical results are Interesting, the more 
Important contributions of his work are methodological. A basic point 
In Ashenfelter's study Is that participation In the program Is not 
random, and specifically that Individuals whose earnings are unusually 
low just prior to participation In training are the ones most likely to 



3 



The 



.... error term Includes a component that Is specific to an 
Individual and that does not vary over time (known as a "fixed-effect" 
and Included to capture differences In unmeasured factors such as 
motivation and ability); a component that Is specific to a time period 
but does not vary over Individuals (to capture economy-wide changes In 
earnings); and a random component that varies over Individuals and time. 
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enter training. This "dip" 1n earnings Introduces statistical bias In 
any model that contains a fixed-effect 1n the error component 1n the 
estimation equation for current earnings. To deal with this problem, 
Ashenfelter capitalizes on the availability of detailed pre- and 
posttralnlng earnings records for participants and nonpartlclpants and 
(In some cases) estimates first-difference models to remove the fixed 
effect, which eliminates the bias under certain assumptions.* The 
results of the first-difference models generally yield somewhat larger 
net Impact estimates than those obtained from the autoregresvlve 
earnings models described above and also Indicate no decline In Impact 
In subsequent years for either men or women. Asht/nfelter also 
emphasizes the Importance of Identifying the structure of the selection 
process for participants. This observation Is extremely Important In 
evaluating program Impacts, because program Impact estimates may be 
very different and potentially misleading If the selection process 
changes from year to year and such changes are not accommodated In the 
evaluation methodology. 

Cooley, McGutre, and Prescott (1979) provide an alternative methodology 
to analyze the HDTA data. A major focus of their study Is whether the 
CHHS data utilized by Ashenfelter and others to draw a comparison group 
for HDTA trainees provide an adequate comparison group. Their results 
Indicate that, for 1969-71 cohorts of Institutional HDTA trainees, 
program "no-shows" (I.e., Individuals who applied to the program but 
did not participate) are a more reliable comparison group than 
Individuals drawn from the CWHS. The basis for their choice Is the 
empirical observation that the autocorrelation coefficients of earnings 
are very similar for the no-show and participant groups In the 
pre-program years, whereas the coefficients differ considerably between 
participants and the CWHS comparison group. 5 

Cooley, McGuIre, and Prescott utilize a simple analysls-of-varlance 
model to estimate the net Impact of HDTA classroom training and 
on-the-job training programs on participants' post-program SSA 
earnings. Unlike most other studies, the estimated net effects of 
training are very similar for men and women, and range between $200 and 
$500 annually, with some Indication that training effects Increase over 
time. The authors also find that the training programs Increased 
earnings of men primarily by Increasing their probability of 
employment, defined as having SSA earnings of at least $50. For women, 
however, the positive Impact on the probability of employment accounts 
for only one-eighth of the total Increase In earnings, suggesting that 



First-difference models are regression equations In which all 
variables (dependent and Independent) are expressed as changes between 
two time periods. This Is often a convenient form for estimating 
program Impacts because It enables one to remove the effects of 
variables that do not change over time. 

5 The autocorrelation coefficients represent the correlation between 
earnings In a given year and earnings In each preceedlng year. Such 
Information Is often useful In understanding how earnings change over 
time. 
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the primary Impact of training for women Is through higher wage rates 
or more hours worked per year. 

Although the Cooley, HcGuIre, and Prescott study Is an important 
contribution to the literature, some potential limitations of their 
analysis must be recognized. For example, although n^-shows appear to 
be a reliable comparison group, at least to the extent of having 
slml lar earnings autocorrelation patterns, no-shows Introduce another 
potential selection bias 1n that they may be systematically different 
from trainees 1n unobserved characteristics that may be responsible for 
the observed program Impacts. It should also be noted that these 
authors' rejection of the CWHS data for drawing comparison groups for 
the 1969-71 cohort of trainees does not Imply that Ashenfelter 1 s use of 
these data to draw a comparison group for the 1964 cohort Is 
Inappropriate. That Is, since the composition of trainees shifted 
drastically between 1964 and 1969 toward a younger, more poorly 
educated, and more disadvantaged group of Individuals, the CWHS data 
could have generated a reliable comparison group for the earlier sample 
of trainees that had more job skills on average. 

A paper by Klefer (1979) analyzes the effect of training on the 
earnings of trainees In four programs: (1) HDTA Institutional Training 
Program, (2) Job Opportunities In the Business Sector (JOBS), (3) the 
Job Corps, and (4) the In-School and Summer Neighborhood Youth Corps 
(NYC). His analysis Is based on a data set developed by the Office of 
Economic Opportunity (0E0) and the Department of Labor (DOL) that 
collected survey data on trainees and members of a comparison group 
from ten standard metropolitan statistical areas (SMSAs) judged to be 
representative of major SMSAs. The comparison group was developed from 
a household screening of a large sample of Individuals In each SHSA who 
were eligible for the program but did not apply, and then matched to 
participants on the basis of age, race, and sex. Klefer presents 
evidence Indicating that the comparison group and the trainee group are 
different 1n Important characteristics, ~ which reduces the validity of 
the program Impact findings. By controlling for earnings history prior 
to program participation, however, he Is able to statistically reduce 
the potential differences between the two groups. 

Klefer f s results Indicate that the effects of training on earnings 
varied considerably by program type, sex, and racial group. For the 
adult training programs, HDTA and JOBS, he concludes that the Increase 
1n annual earnings for women 1s approximately $500 and 1s somewhat 
higher for nonblack women. The Impact of training on the earnings of 
adult men Is less clear. Klefer finds that men trained through HDTA or 
JOBS generally did not regain the earnings position they held prior to 
training relative to the comparison group, although black men who 
participated 1n the JOBS program are estimated to have obtained modest 
earnings gains of approximately $100 to $150 per year. 

In a companion paper that focuses exclusively on the male HDTA 
classroom training sample of the 0E0/D0L data base, Klefer (1978) 
estimates the effect of training separately on employment and weekly 
earnings. His model estimates the effect of training as a nonlinear 
function of the number of weeks an Individual stays 1n the program 
(I.e., the number ot weeks and the number of weeks squared are included 
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as Independent variables In the regression equation). Klefer's 
estimates Indicate no effect of training on the probability of 
employment for either black or white men and negative Impacts on weekly 
earnings for both groups, although the Impact 1s only significant for 
blacks. Although these negative net Impact findings may result partly 
from the comparison group problem Indicated above, they are not 
Inconsistent with results from recent evaluations of CETA described 
later In this chapter. 

A recent paper by Gay and Borus (1980) provides an interesting 
reanalysls of data from the 0E0/D0L sample. They estimate separate 
earnings equations for race/sex subgroups for each program and then 
attempt to Identify short-term economic Indicators that are capable of 
predicting participants' long-term post-program earnings ga1ns--an 
Important policy question. The long-term Impact of program 
participation Is measured by comparing the 1973 SSA earnings of the 
participant and comparison group members. The beginning of 1973 
corresponds to approximately one-and-one-half to three years after 
participants left the program. The average earnings Impacts reported 
by Gay and Borus for participants In the adult training programs (HDTA 
and JOBS) are comparable to those found by Klefer and Indicate that 
women (particularly nonblack women) benefited most from program 
participation, and that the Impact of the youth programs (.lob Corps and 
NYC/OS) on earnings are negative. 

Gay and Borus' attempt to Identify reliable short-term Indicators of 
long-term program performance, however, was unsuccessful. 
Specifically, they conclude that short-term Indicators such as 
"placement" are not reliable predictors of long-term program Impacts. 
It should be note", however, that their measure of various performance 
Indicators depends only on the mix of clients served. Although the mix 
of clients served Is likely to be an Important determinant of variation 
1n service-provider effectiveness, It Is also Important to adjust the 
performance Indicator for differences In local labor market 
conditions. As such, their conclusion that placement Is not a reliable 
predictor of long-term earnings Impacts should be Interpreted with 
caution. 



Schiller (1978) reports a longitudinal analysis of the Impact of WIN on 
participants' post-program earnings and welfare dependency. This study 
attempts to Identify the specific dimensions of relative effectiveness 
by explicitly examining the Interaction of program activity and 
economic conditions. The study compares the earnings and welfare 
dependency of WIN participants with that of a comparison group of 
nonactlve registrants to evaluate the short- and long-term Impacts of 
the program and to determine the cost-effectiveness of WIN. Because 
nonactlve registrants are likely to differ systematically' from WIN 
participants on characteristics that affect program outcomes, however, 
the study results should be Interpreted with caution. 

Schiller distinguishes five levels of treatment provided by WIN- (1) 
no service; (2) advice and assistance In Job placement; (3) classroom 
education; (4) vocational training; and (5) assignment to on-the-job 
training or public service employment. He finds modest earnings gains 
and reductions In welfare dependency as a result of program 



participation, although effects vary considerably with part^cloant 
characteristics and the services provided. His results Indicate that 
men benefited only from OJT or subsidized public employment, whereas 
women benefited from all of the services. In addition, his results 
Indicate that Individuals without recent employment experience 
generally showed the largest gains. 

A major experimental evaluation of a job training program was conducted 
as part of the Supported Work Demonstration, a transitional work 
experience program that provided training In basic work skills. The 
demonstration offered supported work jobs to a random sample of 
eligible individuals In four major hard-to-employ groups: 
ex-offenders, ex-addicts, high school dropouts, and women who were 
long-term recipients of AFDC payments. The unique feature of the 
Supported Work concept was the provision of a subsidized but productive 
job In a setting featuring peer support, close supervision, and a 
gradual Increase of work pressure and demands on the job over time. In 
addition, the experimental design used allows one to overcome the 
problem of selection bias and obtain valid estimates of the Impacts of 
program treatments. 

Evidence from the Supported Work Demonstration indicates that of the 
four target groups enrolled in the study, long-term recipients of AFDC 
benefited the most (Masters, 1981). In particular, participation in 
Supported Work for the AFDC group led to an increase in their 
employment rate, hours worked, and earnings, both While they were In 
the program and after they left it. Concurrent with these positive 
earnings effects was a net reduction In welfare dependency for those in 
the experimental AFDC group. There was also some indication that the 
program Improved the quality of employment of the experimental 
population. 6 The program was not generally successful for 
disadvantaged youths or other predominantly male target groups, 
although there were positive earnings Impacts for former drug users who 
had the least employment opportunities (Dickinson, 1981). 

The literature evaluating the effects of employment and training 
programs for disadvantaged youths has also grown considerably in recent 
years. In addition to the Supported Work Jemonstratlon, and the 
studies described above that evaluated the youth programs contained in 
the 0E0/D0L sample, there have been major evaluations of the Job Corps 
and the Neighborhood Youth Corps (NYC), among others. Because these 
programs have several elements In common with JTPA youth programs, we 
briefly review what is known about them below. 

The most in-depth evaluation of the impact of the Job Corps program was 
conducted by Mallar et al. (1982). The Job Corps provides economically 



Although these results appear quite promising, their 
generallzabillty is likely to be limited* This is primarily because 
the sample of lon^-term AFDC recipients who volunteered for the 
demonstration and were subsequently assigned to experimental/control 
status is not representative of AFDC recipients overall. For example, 
Masters (1981) estimates that only 17 percent of all AFDC recipients 
met the Supported Work eligibility criteria. 



disadvantaged youths between 16 and 21 years of age with basic 
education, vocational training, and support services within a 
residential setting 1n order to improve their earnings potential and 
help them become better citizens. As such, Job Corps programs have 
several elements 1n common with JTPA programs for disadvantaged youth, 
although corpsmembers are generally much more disadvantaged and the 
residential living feature 1s a major distinguishing characteristic. 

The methodology used 1n the Hallar study Involves comparing the 
post-program experiences of a sample of Job Corps participants with the 
experiences of a sample of disadvantaged youths selected from 
comparison sites. The comparison group of youths was developed from 
lists of school dropouts (70 percent) and from applicants to local 
Employment Service offices (30 percent). Based on detailed survey data 
collected over a four-year post-program period, the findings Indicate 
that corpsmembers worked an average of three additional weeks per year 
and had higher earnings than nonpartldpants by approximately $600 per 
year. The overall effects appear to persist throughout the four-year 
observation period. The Hallar study results also Indicate that men 
and women without children consistently benefited more 1n terms of 
employment and earnings than did women with children. In fact, 
participating women with children were employed less 1n the 
post-program period than otherwise similar comparison group members. 
The Hallar study also finds that program completers consistently 
benefited more than noncompleters 1n terms of employment and earnings 
and that early dropouts from the Job Corps benefited little or not at 
all . 



Although the general approach used 1n the Hallar study 1s quite 
reasonable, the empirical results should be Interpreted with caution 
for several reasons. First, because ES applicants are likely to be 
more job ready on average than high school dropouts, such a mixed 
comparison group strategy 1s not likely to result 1n unbiased Impact 
estimates. Second, the Initial sample design called for a 
cross-section of participants at a point 1n time, which systematically 
excludes a large fraction of short-term participants. For example, 
40 percent of all enrollees 1n FY 1977 (the year 1n which the Hallar 
sample was drawn) dropped out of Job Corps during the first 90 days, 
compared with nine percent of corpsmembers 1n the Hallar study. Both 
of these procedures create an analysis sample that 1s unrepresentative 
of the population of Job Corps enrollees, which necessarily limits the 
general1zabH1ty of the net Impact estimates. Furthermore, the quality 
of the comparison group developed for young women was particularly 
deficient; females 1n the comparison group were much more likely to be 
married or to have children. Because of the well -documented evidence 
that marital status and presence of children affect the work effort of 
women, the Hallar study's Inability to match well on marital status for 
women creates additional biases 1n assessing program Impacts for this 
subgroup. 

The Neighborhood Youth Corps Program (NYC) was created by the Economic 
Opportunity Act of 1964 to provide part-time work experience, remedial 
education, and Job training assistance to disadvantaged youths who did 
not complete high school or were likely to drop out of high school. In 
addition to a summer employment program, NYC had both an 1n-school 
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program (providing part-time employment while attending school to 
encourage youths to remain 1n school and graduate) and an out-of-school 
program (providing work experience and some vocational training). The 
In-school component of NYC was examined by Somers and Stromsdorfer 
(1970), and the out-of-school component was examined by Borus, Brennan, 
and Rosen (1970) . 

The Somers and Stromsdorfer study Is based on a nationwide sample of 
S76 Individuals who enrolled In In-school and summer NYC programs In 60 
areas between July 1965 and June 1967. For a comparison group, the 
authors used students who met the NYC eligibility requirements but did 
not participate In the program. After controlling for differences In 
measured characteristics through multiple regression techniques, they 
find that the NYC In-school program had a significant Impact on the 
earnings of participants after high school. The program had no effect 
on the wage rates of participants; rather, the earnings gains were a 
result of Increased employment of participants. Benefits to black 
women accounted for much of the overall gain for the total sample. 
These results should be Interpreted with caution, however, because 
statistical tests provided by the authors Indicate that there were 
systematic differences between the participant and comparison samples. 
Although the regression analysis method used by the authors In part 
corrects for this problem, at least for measured variables, biases may 
remain because of differences In unmeasured characteristics (e.g., 
ability, work attitude) that affect post-program outcomes. 

Borus, Brennan, and Rosen (1970) evaluated the out-of-school NYC 
program using data on 604 participants In five urban areas In Indiana. 
Participants in this study are defined as Individuals who had worked at 
least one day and had left the program by December 31, 1966. 
Post-program earnings of participants (obtained from UI earnings 
records) are compared with those of a comparison group of 166 
individuals who applied for the program and were eligible but did not 
enroll because (1) they were placed on a waiting list and were never 
called, (2) they could not be reached for a job assignment, or (3) they 
did not report when assigned to a job. 

In contrast to the Somers and Stromsdorfer study of the NYC in-school 
program, as well as to most other evaluations of employment and 
training programs, the authors find no earnings gains for female 
participants and relatively small earnings gains as a result of NYC 
participation for young men. These unexpected results could In part be 
due to lack of comparability between participants and comparison group 
members. There is some evidence that individuals with less prior 
schooling benefited more from the program and that length of 
participation Is positively correlated with post-program earnings gains. 

Another program that was designed to help disadvantaged youths was the 
Concentrated Employment Program (CEP). CEP provided youths with work 
experience and job search assistance. The only major attempt to 
evaluate CEP, by Klrschner Associates (1969), compared participants 1 
behavior before and after program participation to Infer program 
Impacts. Although such a design can provide useful Information on 
gross program Impacts, the large post-program employment and wage 
Increases reported should not be Interpreted as measures of net program 
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Impacts. The limitations of a before-after comparison design for 
measuring program Impacts are particularly severe for youths, who have 
steep age-earnings profiles and would have had considerably higher 
earnings at later periods even without participating 1n the Droaram 



EVALUATIONS OF CETA PROGRAMS 



The recent availability of data from the Continuous Longitudinal 
Manpower Survey (CLMS) has made It possible to evaluate the net Impact 
of CETA programs. The CLMS collected extensive Information on the 
socioeconomic and demographic backgrounds, the In-program experiences, 
and the subsequent labor market experiences of a nationally 
representative sample of CETA participants beglnnlno In FY 1975. To 
measure net program Impacts, Social Security Administration (SSA) 
earnings records (both pre- and post-program) have been attached to the 
CLMS public use files anr" A - .--«-— ... - . - - . . 
(CPSs), which have been 
recently, most net Impact 

been conducted by Westat, Inc. In this sectlorTwV describe" the ^We'stat 
studies and other recent net Impact evaluations of CETA using the CLMS 
data. 



pre- and post-program) have been attached to the 
and to various March Current Population Surveys 
?en used for drawing comparison groups. Until 
act evaluations of CETA based on these data have 



Through a series of studies, Westat has examined the post-program 
earnings Impacts for FY 1975, FY 1976, and FY 1977 CETA enrollees who 
participated In a program activity for at least eight days. Because 
their basic approach and findings are quite similar across these 
reports, we discuss below their methodology and findings as described 
In their July 1982 report. In that report, Westat presents Impact 
estimates for the first and second post-program years (1977 and 1978) 
for FY 1976 enrollees who terminated by December 31 , 1976. Their 
analysis Is based on a comparison of the Social Security Administration 
earnings records of CETA participants and those of a matched comparison 
group drawn from the March 1976 CPS. 7 Their net Impact estimates are 
derived from weighted regression equations. 8 

Westat' s results Indicate that CETA termlnees earned approximately $300 
more In both 1977 and 1978 than Individuals In the matched comparison 
group. There appear to be considerable differences In Impact by 
program activity, with OJT trainees experiencing the largest earnings 

7 The comparison group was developed using stratified matching 
techniques separately for low, medium, and high pre-program earners 
The highest priority matching variables for low earners were 
demographic characteristics (e.g., sex, race/ethnicity, age, 
education), whereas, prior earnings was emphasized In the matches for 
both medium and high earners. 

8 CPS cases In each cell were weighted so that the sum of the 
weights equalled the number of CLMS cases In that cell, and where cells 
were defined by combinations of characteristics on which the match was 
based. The weighting was done to create a comparison group that was 
dlstrlbutlonally more similar to CETA participants on background 
variables believed to be associated with earnings potential. 

16 

ERIC 23 



gains, followed by classroom training and PSE trainees. Individuals 
enrolled In work experience programs had lower post-program earnings 
than Individuals 1n the comparison group, although the differences are 
not statistically significant. Westat also finds that the Impacts were 
much larger for Individuals who remained 1n the program longer. In 
terms of subgroup effects, Westat 1 s results Indicate that women 
experienced larger earnings gains than men, and that the effects for 
minority men were particularly small. They also observe a general 
pattern of Increasing earnings gains with age, with generally negative 
estimates of program impacts for youths. 

In interpreting the implications of Westat's findings, one should use 
caution for several reasons. In particular, the procedures used to 
align and match the CLMS and CPS samples and the specific sample frame 
and sample exclusion decisions followed have certain disadvantages. In 
addition, the specification of the weighted regression model to 
estimate net impact was inappropriate. Because many of the recent 
evaluations of the impact of CETA adopted Westat 1 s methodological 
choices on these issues, below we Indicate some of the limitations of 
their approach. 

Westat drew comparison groups using a stratified matching technique. 
The stratified technique, which selects several variables and divides 
each sample into cells based on factorial combinations of those 
variables, has several potential disadvantages. First, higher-priority 
variables must be matched exactly, even if this results in a very large 
difference in a lower-priority variable. Second, only categorical 
variables can be used in stratified matching. Because the 
categorization of a continuous variable is necessarily arbitrary, 
individuals who are close on a continuous variable may be judged to be 
far apart when categorized (e.g., if an age cell included individuals 
30 to 44 years of age, an individual aged 44 could be matched to 
another aged 30 but not to one aged 45). Third, very few variables can 
be used to match the samples. Fourth, a very large number of potential 
comparison group members is required to generate exact matches. 
Despite these potential deficiencies, evidence presented in Dickinson, 
Johnson, and West (1986) indicates that the net impact estimates 
obtained from stratified matched comparison groups are quite similar to 
the results obtained when more sophisticated matched comparison groups 
are used. 

A second potential problem concerns the way in which the CLMS and CPS 
samples are aligned. Westat divides the CLMS sample into cohorts based 
on the fiscal year in which participants enrolled in CETA and matches 
these individuals with CPS sample members who were interviewed in March 
of that fiscal year. For example, the FY 1976 CLMS cohort (individuals 
who enrolled in CETA between July 1, 1975 and June 30, 1976) is matched 
with individuals in the March 1976 CPS. The way in which the samples 
are aligned is important because SSA earnings, which are used to 
measure the impact of CETA and are also the primary matching variables, 
are defined on a calendar year basis. Given that SSA earnings are 
measured on a calendar year basis, it is very difficult to match well 
on the pre-program decline in earnings experienced by CETA participants 
when using a fiscal year alignment, without introducing additional 
analytical complications. For example, Westat matches on calendar year 
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l«JL*?f?i earp : 1n 9 s rather than rcatch1n 9 on 1974 SSA earnings and 
potentially miss matching on the pre-program decline 1n earnings 
experienced by CETA participants. The procedure of matching on 1975 
SSA earnings results 1n biased estimates of the Impact of CETA 
however, because 1975 earnings Includes up to six months of 1n-program 
earn ngs, which may be particularly low (e.g., for classroom training 
participants) or high (e.g., for PSE participants) and even Includes 
post-program earnings for some participants. It should be noted that 
the change (1n 1977) to an October 1 to September 30 fiscal year should 
reduce the bias Introduced by matching on postenrol lment earnings 1n 
analyzing more recent fiscal year cohorts of CETA enrollees, although 
the problem must be dealt with 1n analyzing the net Impact of JTPA 
because program year cohorts are defined on a July 1 to June 30 basis. 

A third Issue relates to the specification Westat uses to estimate 
their weighted regression Impact model. Specifically, although the 
participants and the comparison group members are matched on 1975 SSA 
earnings, this variable 1s not included as a control variable 1n the 
weighted Impact regression model. Because the weights are highly 
correlated with 1975 SSA earnings and because 1975 SSA earnings are 
highly correlated with post-program SSA earnings, the weights are 
correlated with the error term 1n the true model for CPS cases. As 
shown In Dickinson, Johnson, and West (1986), this procedure leads to 
overstating the Impact of CETA. 

A fourth Issue concerns decisions regarding the Individuals to be 
Included 1n the sample from which the comparison group 1s selected 
(I.e., sample frame decisions). An Important CPS sampling frame Issue 
Involves restrictions to help ensure that the comparison group more 
closely matches CETA participants 1n terms of pre-program labor market 
experience or attachment. Such restrictions are necessary because many 
CLHS sample members (particularly women and youths) had little recent 
labor market experience, yet, by definition, were 1n the labor force 
when they applied to CETA. In an attempt to ensure that the comparison 
groups match participants on pre-program labor market attachment, 
Westat excludes from the CPS Individuals who were not 1n the labor 
force during the Interview week, unless they had worked part of the 
previous year. However, U 1s possible that such a sample criterion 
Includes 1n the comparison group Individuals who have dropped out of 
the labor force and who are much more likely to continue out of the 
labor force 1n the post-program period. This would lead to an upwardly 
biased estimate of CETA's net Impact. 

A final Issue concerns decisions to exclude certain cases from the 
analysis (I.e., sample exclusion decisions). The major sample 
exclusion Issue for CLMS cases Involves restrictions on length of stay 
1n CETA. Westat restricted their analysis to Individuals who 
participated 1n CETA for at least eight days. Although CETA clearly 
cannot have a large Impact on those who participate only a few days, 1t 
1s possible that excluding participants who stayed fewer than eight 
days (14 percent of the trainee sample) could have Introduced further 
selectivity biases Into the analysis 1f short-term participants 
differed from other participants on unmeasured characteristics, such as 
motivation or Job readiness. Evidence presented 1n Dickinson, Johnson, 
and West (1986) Indicates, however, that the decision to exclude 
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Individuals who stayed fewer than eight days does not affect Westat's 
net Impact estimates, 

A recent study by Dickinson, Johnson, and West (1986) estimates the net 
Impact of CETA on participants 1 post-program earnings using an approach 
that differs from Westat In several respects. Their Impact estimates, 
using 1978 SSA earnings as the outcome measure, are obtained from a 
CLHS sample of participants that enrolled In CETA during calendar year 
1976 and that terminated by December 31, 1977. Separate matched 
comparison groups are drawn overall and by program activity for adult 
men, adult women, young men, and young women using a nearest-neighbor 
matching technique based on a modified Mahalanobls distance metric that 
has several advantages over stratified matching, 9 

In general, Dickinson, Johnson, ana West find that Individuals who 
enrolled In CETA In 1976 do not have significantly better post-program 
employment experiences than comparable Individuals In the matched 
comparison groups. This conclusion Is based on evidence of 
statistically significant but negative earnings Impacts for both adult 
and young mtn, and modest but not statistically significant earnings 
gains for adult and young women. Adult women 1n PSE programs and young 
women In OJT programs are estimated to have experienced significant 
earnings gains; negative program effects are estimated for work 
experience participants for all age-sex subgroups. The pattern of 
large negative Impacts for men and relatively small positive Impacts 
for women Is Invariant to the use of alternative procedures and 
assumptions. 10 It should be noted that the negative Impacts for men 
are not unreasonable If CETA diverted male participants from productive 
job search, and men might have continued to enroll In CETA despite the 



The nearest-neighbor technique calculates a distance between an 
Individual In the CLMS and each Individual In the CPS, based on a 
number of dimensions, and then matches Individuals in the CLMS to the 
closest Individual in the CPS. The Mahalanobls distance metric 
essentially determines how many standard deviations apart Individuals 
are on each variable, adjusting for the observed covariances of the 
matching variables, and requires that individuals be closer on 
variables with smaller standard deviations and allows individuals to 
be farther apart on variables with greater standard deviations. The 
Mahalanobls metric has been used extensively In developing matched 
comparison groups. See Cochran and Rubin (1973), and Rubin (1979). 

10 Dickinson, Johnson, and West (1984a, 1984b) provide considerable 
evidence that the negative Impacts of CETA on SSA earnings for men 
cannot be attributed to potential data deficiencies. Specifically, 
these negative Impacts are due neither to the omission of uncovered 
earnings from the outcome measure, nor to the cap on SSA earnings or to 
potential contamination of the comparison group to the extent It 
contains some unknown proportion of CETA participants. Moreover, the 
results from alternative econometric models that attempt to correct for 
potential selection basis (I.e., differences in unmeasured 
characteristics between CLMS and CPS sample members) provide support 
for the negative impact findings for men and provide some evidence that 
the true impacts for adult women could be somewhat larger. 
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negative post-program effects because of substantial training subsidies 
and 1n-program earnings. In fact, consistent with this hypothesis, 
Dickinson, Johnson, and West (1984a) report positive post-program 
Impacts of CETA for adult men who did not receive stipends. 

Although the overall Impacts reported by Dickinson, Johnson, and West 
are Interesting and suggest that early CETA programs were not very 
effective In Increasing the earned Income of participants, the more 
Important contribution of their study lies In their detailed 
examination of the sensitivity of CETA net Impact estimates to 
alternative methodological procedures , In a recent summary paper, 
Dickinson, Johnson, and West (1985) compare the methodologies and 
resulting net Impacts of several of the recent CETA studies discussed 
in this chapter. They find that the range of overall CETA impacts 
varies from roughly -$300 to +$300 and that these differences largely 
can be explained by the different methodological procedures used. 
Although it Is comforting that the differences In net Impact estimates 
can be accounted for by the different methodological procedures used, 
it Is troublesome that the range of estimates Is so large. Below we 
describe the various methodological procedures that appear to affect 
net Impact estimates. 

Dickinson, Johnson, and West find that CETA net Impact estimates are 
quite sensitive to whether individuals without recent labor market 
experience are Included In the comparison groups. In particular, they 
demonstrate that including individuals with less attachment to the 
labor force (e.g., disabled, or dropped out of the labor force, or 
never worked) in the comparison groups causes estimated program Impacts 
to increase. 11 They also demonstrate that CETA Impacts are sensitive 
to the alignment of the CPS and CLHS samples, and in particular they 
find that estimated impacts are more positive for early enrollees for 
whom the preprogram decline In earnings Is more accurately measured. 
This Indicates the Importance of developing matched comparison groups 
on a quarterly or semi-annual basis in order to avoid biases due to 
misal ignment . 

Dickinson, Johnson and West also provide considerable evidence on the 
sensitivity of estimated program Impacts to the matching technique 
used. An important finding of their study is that the estimated 
impacts are quite robust to the matching procedures used. 
Specifically, they obtain similar net Impact estimates using either 
their overall nearest-neighbor matched groups, their closely matched 
nearest-neighbor by-program matched comparison groups, or using the 
large (unmatched) CPS ellgible-for-match sample. Moreover, based on a 
detailed replication of Westat's analysis, they find that when other 
methodological factors are treated comparably, one obtains similar 

11 The main approach used by Dickinson, Johnson, and West (1986) 
requires adults In the CPS to be in the labor force in the survey week 
to be eligible for Inclusion in the matched comparison groups. It 
should be noted that this is a much more stringent requirement than 
Westat's decision to include in the comparison groups individuals who 
worked in the previous year but were out of the labor force during the 
Interview week. 
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estimated net Impacts using either the nearest-neighbor matches or 
Westat's stratified cell matches. Thus, the estimated Impacts do not 
appear to depend on the actual matching technique used. This has 
Important Implications for future analyses. 

Bassl (1983, 1984) builds on the methodological work by Ashenfelter and 
uses the matches developed by Westat to estimate the net Impact of CETA 
for white women, minority women, and minority men who were at least 23 
years old when they enrolled 1n CETA In FY 1976. Her results Indicate 
that a simple fixed-effects estimator Is sufficient to eliminate the 
bias Introduced by nonrandom selection for all subgroups except white 
men, although there seem to be substantial differences In pre-program 
earnings between high-Income trainees and their matched comparison 
group. 12 Because of the lack of an adequate comparison group for 
white men, she does not estimate the Impact of CETA on this subgroup. 
Her overall findings Indicate that women experienced substantial 
earnings gains, with gains for white women somewhat larger. Although 
Bassl's fixed-effects estimator Is a reasonable approach provided the 
underlying assumptions are satisfied, It appears that her results are 
biased upward because the sample Is not limited to Individuals who 
terminated by December 31, 1976, and thus the Impact estimates for 1977 
and 1978 Include In-program earnings for CLMS cases. This 1s a 
particularly severe problem for PSE participants. 

A recent report by Bassl et al. (1984) analyzes CLMS data for FY 1977 
CETA enrol lees to estimate the Impact of employment and training 
programs on two key target groups for JTPA, namely youths and the 
economically disadvantaged. To estimate the Impact of CETA on the 
post-program earnings of youths, the authors use matched comparison 
groups developed separately for each program activity. For their 
analysis of the Impact of CETA programs for economically disadvantaged 
adults, they Include In the comparison group all Individuals from the 
March 1977 CPS who were economically disadvantaged. Although evidence 
pres?nted on the reliability of the comparison groups Indicates that 
considerable creaming may have occurred, the authors argue that a 
fixed-effects estimator Is generally sufficient to obtain unbiased 
Impact estimates. 

Using both a fixed-effects and a random-effects model, and with 
different base years, Bassl etal. find women have larger earnings 
gains than men, and that PSE and OJT result In the largest earnings 



If the error term 1n the Impact model contains a fixed-effect 
(I.e., an unobserved component that 1s specific to an Individual and 
does not vary over t1me--perhaps reflecting permanent differences 1n 
ability or motivation) that Is correlated with other exogenous 
variables, then ordinary least squares estimates of a standard Impact 
model are generally biased. The "fixed-effects estimator," or 
"first-difference estimator," suggested by Ashenfelter Involves 
differencing the model over time (from the pre- to post-program period) 
to purge the error term of the fixed component. The fixed-effects 
estimator will produce unbiased program Impact estimates provided 
certain assumptions are met (see Chapter 5). 



gains. Id The large positive Impact for PSE programs, however, 
appears to be 1n part a result of Including 1n the analysis sample 
Individuals who were still participating 1n PSE programs during the 
outcome year. Results for youths follow the same relative pattern, 
although the overall Impact of CETA programs on earnings appears to be 
negative for youths. 

Another recent evaluation of CETA programs using a subset of the CLHS 
data base was conducted by Bloom and McLaughlin (1982). Their analysis 
focuses on adults (over 24 years of age) who entered either classroom 
training, on-the-job training, or work experience programs between 
January 1975 and June 1976, and who stayed In the program for more than 
seven days. They compared the post-program Social Security earnings of 
participants with the earnings of a comparison group drawn from the 
March 1976 CPS. Their findings Indicate large positive program effects 
for women of between $800 and $1 ,300 per year that are similar across 
type of program, and small but statistically Insignificant earnings 
gains for men. 

Although these results are consistent with the general pattern that 
women ggln more from CETA programs than men, the unusually large 
program Impact observed for women, as well as the atypical result of 
similar Impacts across programs, appears to be the result of two 
problems In their approach. First, no «- tempt was made to ensure that 
Individuals Included In the comparison group were In the labor force at 
the same time that Individuals 1n the CLMS sample were enrolling In 
CETA. By Including In the comparison group a substantial proportion of 
Individuals who were out of the labor force (particularly for women), 
the results can severely overstate the Impact of CETA. A second 
problem concerns the Inclusion In the estimation model of a time-trend 
term that extrapolates the pre-program decline In earnings for 
participants Into an expected further decline In the post-program 
period. A comparison of actual post-program earnings with expected 
earnings based on a trend term could result 1n large estimated Impacts 
even If trainees' earnings did not return to their pre-program level. 
A third problem concerns the decision to exclude from the comparison 
group any person who earned the Social Security taxable maximum In any 
of the years from 1970 to 1975. This systematically excludes 
comparison group members with relatively high earnings, particularly 
those whose earnings were Increasing just before the program, and given 
the Inclusion of the trend term In the model, also results In an 
upwardly biased estimate of program Impacts. 

In a recent paper, Geracl (1984) uses Westat's matched comparison 
groups for FY 1976 CETA enrollees and estimates the Impact of CETA on 
post-program earnings for adults separately for men and women. The 
specification allows the net impact to vary within program activity by 
race, age, and length of stay. The post-program outcome measure Is the 



A "random-effects" model assumes that the error components In the 
regression equation are Indeed random (I.e., the fixed-effect Is not 
correlated with other exogenous variables In the model). If this Is 
the case, ordinary least squares techniques will produce unbiased net 
Impact estimates. 
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average of SSA earnings from 1977 to 1979, and the model controls for 
average earnings from 1972 to 1974 and other demographic 
characteristics. Geracl finds significant earnings gains for adult 
women, but generally Insignificant or negative net gains for men. 
Given that the net Impact model Is estimated by weighted least squares 
and does not control for SSA 1975 earnings (one of the key matching 
variables), however, this causes the estimated program Impacts to be 
upwardly biased, as Indicated earlier. 

In addition to estimating net program Impacts, Geracl also examines 
alternative short-term Indicators of long-term earnings Impacts. This 
analysis 1s motivated by JTPA's focus on long-term outcomes and the 
recognition that, since program management decisions regarding rewards 
and sanctions cannot wait for definitive long-term results, It Is 
Important to Identify short-term Indicators that are reliable proxies 
of long-term program Impacts, Geracl tests several measures of 
short-term Indicators, Including Immediate outcomes such as whether 
placed at termination and the wage rate at termination If placed, as 
well as short-term post-program measures (measured alternatively at 
three, six, or nine months after termination) such as the proportion of 
time employed, the proportion of time In the labor force, the average 
wage rate, and total earnings. 

Unlike Gay and Borus, Geracl finds that placement status at termination 
1s significantly correlated with long-term post-program earnings gains 
for both adult women and adult men. Like Gay and Borus, however, 
GeracTs results contain a potential statistical bias because he was 
unable to adjust for differences In local labor market conditions. His 
results also Indicate that although the correlation between short-term 
post-program Indicators and other measures of long-term post-program 
outcomes (e.g., post-program earnings, pre-post changes In earnings, 
and Indirect net Impact gains) Increases considerably as the length of 
the follow-up period Is Increased from three to nine months, there Is 
little change 1n the correlation between these Indicators and r>et 
Impact measures as the length of the follow-up period Increases. 
Although this suggests that reasonably valid Indicators of long-term 
net Impacts could be obtained from follow-up data three to six months 
after termination, 1t would be Important to obtain slml lar results 
after adjusting for differences In local labor market conditions before 
using his findings to make decisions on the appropriate length of 
follow-up. 

Although all of the recent CETA net Impact evaluations have been based 
on comparison groups developed from the CPS, the usefulness of the CPS 
as a source for drawing matched comparison groups has been challenged 
In a recent study by Fraker and Haynard (19B4). Fraker and Haynard 
draw several different comparison groups from the CPS separately for 
youths and AFDC recipients who participated 1n the Supported Work 
demonstration and compare the resulting net Impact estimates tc those 
obtained when the true control group 1s used. Because the net Impact 
estimates obtained from various C PS-based comparison group 
methodologies are generally quite different from the estimated Impacts 
when the true control group Is used, Fraker and Haynard conclude that 
comparison group methodologies should be used with extreme caution and 
that limitations of the CPS are In part responsible for the fallings of 
this approach. 

23 



In determining the Implications of the Fraker and Haynard study. It Is 
Important to keep In mind that their results are based on a very, tough 
test of the general usefulness of the CPS for selecting comparison 
groups for evaluating the Impact of employment and training programs. 
By design, Supported Work participants differ considerably from 
participants In other employment and training programs In that they are 
more likely to be high school dropouts, less attached to the labor 
force, and more Involved In criminal activities. For example, nearly 
60 percent 0 f the youths enrolled In Supported Work had been arrested 
and nearly 40 percent had been convicted of criminal activities 1n the 
preenrollment period, whereas only five percent of CETA participants 
had been previously arrested. Because of the unique attributes of 
Supported Work participants, Fraker and Maynard are unable to develop 
comparison groups from the CPS that match experimental well on 
measured characteristics. 

Given that the various CPS-based comparison groups do not generally 
match Supported Work experimental even on measured characteristics, 1t 
1s not surprising that the net Impact estimates obtained when these 
groups are used often differ from the Impact when the true control 
group Is used. In fact, given the unique attributes of Supported Work 
experimental, It Is very reassuring that CPS-based comparison groups 
tend to yield valid Impact estimates for AFDC women, which are a very 
difficult group to match. Their conclusion that the CPS Is not a good 
source for drawing matched comparison groups for Supported Work youths 
1s 1n part a result of the Inability to match youths on measured 
characteristics (e.g., criminal behavior) and may also In part be due 
to the difficulty of developing reliable comparison groups for youths 
In the absence of detailed pre- and post-program data on schooling 
behavior. 14 



REGENT METHODOLOGICAL DEVELOPMENTS 



As the empirical literature described above has developed, there has 
been an accompanying methodological literature focusing on procedures 
that can be used to correct for various selection biases to obtain 
unbiased estimates of program net Impacts. This literature Includes 
the Ashenfelter paper discussed above, recent work motivated by the 
Ashenfelter paper, and other papers that address selection bias Issues 
1n a more general context. In this section we briefly describe some of 



The l ar ge negative net Impact estimates obtained for youths when 
each of the CPS-based matched comparison groups Is used, which are not 
obtained when the randomly assigned control group Is used, may also be 
due to an alignment mismatch. Specifically, because approximately 
85 percent of the youths In Supported Work were enrolled after March 
1976 and because more than half of the cases eligible to be Included In 
the comparison groups came from the March 1976 CPS, this alignment 
causes matched comparison group members to experience the pre-program 
dip 1n earnings much earlier on average than the experlmentals. As a 
result, regression to the mean would cause the comparison group to 
recover their earnings position earlier than the experimental group and 
bias the Impact estimates downward for youths. 
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the more relevant methodological developments that may have some 
applicability to evaluating the net Impact of JTPA. Because of the 
Inherently technical nature of this discussion, uninterested readers 
may choose to skip this section and proceed directly to our summary of 
the literature and Its Implications. 

The general literature on selection bias has developed from a .oncern 
to obtain statistically unbiased estimates of program treatment effects 
within a nonexperlmental design. As we Indicated above, In the absence 
of a randomly assigned control group, It Is necessary to statistically 
control for all characteristics that Influence the likelihood of 
selection Into the treatment group when measuring program effects. If 
all variables that affect selection Into the treatment group are 
Included In the regression equation that measures program Impacts, or 
If those omitted are unrelated to the outcome, then one would obtain an 
unbiased measure of the treatment effect. If one has not properly 
accounted for the variables (observed or unobserved) that affect 
selection Into the treatment group, then the treatment variable will be 
correlated with the error term In the Impact equation and ordinary 
least squares estimates will be biased. 

To solve the problem of the correlation between the outcome measure and 
the error term 1n the regression equation, Instrumental variable 
procedures can be used. Papers by Heckman (1979), Maddala and Lee 
(1976), and by Barnow, Cain, and Goldberger (1980) provide Instrumental 
variable approaches that can produce consistent estimates of program 
Impacts under certain circumstances or assumptions. Each procedure 
requires the creation of a predicted value of program participation 
from a prior problt equation and Inclusion of either the predicted 
program participation variable, the Inverse of Mills 1 ratio, or another 
constructed variable based on the problt results, in the earnings 
regression. 15 Although these Instrumental variable procedures 
produce consistent estimates of program Impacts, they are only useful 
if one can find variables that affect participation In the treatment 
9roup but do not affect earnings or other outcome variables of 
Interest. Unfortunately, In most cases the Instrumental variable 
created will be correlated with other variables in the equation, making 
it extremely difficult to obtain precise estimates of treatment effects. 

Perhaps a more useful approach to solving selection bias problems Is to 
take advantage of the longitudinal nature of the data to control for 
potential differences between the treatment and the comparison groups. 
Such an approach builds on Ashenfelte^s generalized difference 
estimator using detailed pretralnlng earnings data from participants 
and nonpartlclpants, as discussed earlier. The notion of using the 
longitudinal nature of the data has also been extended by Heckman and 
Robb (1982, 1985). For example, Heckman and Robb (1982) demonstrate 
that a difference estimator that Is symmetric about the year In which 
the decision to enroll In training 1s made can provide u r b1ased 
estimates of program Impacts even when the random component of earnings 



For a description of the problt technique see Finney (1964). The 
construction of the Inverse of Mills 1 ratio Is described 1n Heckman 
(1979) . 
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is correlated over time. That 1s, 1f certain assumptions regarding the 
program participation decision and the error term in the earnings 
equation are met, one can obtain unbiased net impact estimates by 
regressing earnings in year s+t less earnings in year s-t on a 
participant dummy variable and the difference in exogenous variables 
between these two periods, where s is the year in which the decision to 
enroll in training is made, and t represents the number of years after 
s for which post-program earnings are being measured. 

As demonstrated in Dickinson, Johnson, and West (1986), net impact 
estimates obtained from the symmetric difference estimator are very 
sensitive to the choice of the decision year. Moreover, it is likely 
that the appropriate decision year differs among participants 
depending on the date of enrollment into training. That is, for 
individuals who enroll early in a given year, one can reliably use the 
prior year as the decision year; for individuals who enroll late in the 
year, it is likely that the enrollment year is the decision year. This 
problem would be less serious if one could measure earnings on a 
quarterly or semi-annual basis. 

The more recent paper by Keckman and Robb (1985) details the 
assumptions required to use various cross-section and longitudinal 
models to estimate the impact of training on earnings free of selection 
bias. This paper is important because it provides a framework for 
comparing the assumptions used by previous researchers that were not 
often stated explicitly. It is likely that some of the differences in 
net impact estimates reported above are due to differences in 
assumptions regarding the underlying structure of earnings, the 
decision rule governing program participation, the time homogeneity of 
the environment or the distribution of unobservables . It is important 
to carefully describe the assumptions underlying the proposed 
state-level net impact model, so that others can evaluate the 
reasonableness of the assumptions and potentially determine the extent 
to which different impact estimates are due to different underlying 
assumptions. 



SUMMARY AND IMPLICATIONS 

In this chapter we have reviewed several recent evaluations of the 
impact of employment and training programs on participants' 
post-program outcomes, as well as briefly summarized a few recent 
contributions to the methodological literature. The results of these 
studies generally indicate large earnings gains for women, particularly 
nonblack women, whereas the effect of employment and training programs 
on the earnings of adult men is less clear. Although almost all 
studies have found the earnings gains of men to be considerably less 
than those obtained by women, several recent evaluations have found 
that male trainees never regain the earnings position they held prior 
to training relative to otherwise comparable nonparticipants . Why did 
men continue to enroll in these employment and training programs? 
Perhaps because of 1n-program earnings and the substantial training 
subsidies offered by CETA. 

These studies have several important Implications for designing a net 
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Impact evaluation model of JTPA programs. Some of the Implications 
concern the conceptual framework to be used (e.g., outcome measures, 
subgroups and treatments to be examined), and others relate to research 
design and data analysis Issues (e.g., Issues concerning comparison 
group selection such as the alignment of samples, the sample frame 
choice and sample exclusion decisions, and the statistical models 
used). Below we briefly summarize these studies and their Implications 
as they relate to developing a conceptual framework for evaluating JTPA 
at the state level. This brief summary serves as a natural 
Introduction to the next chapter, which discusses the conceptual 
framework for a state-level net Impact model. The Implications of 
previous studies concerning research design and analysis Issues are 
discussed In Chapters 4 and 5 as appropriate. 

In Table 1 we summarize several key features of the employment and 
training Impact evaluations described above. In particular, we list 
the outcome measures examined, the participant groups selected, the 
subgroups of trainees for which separate net Impact models were 
estimated, and the ways In which the treatments were Included In the 
models. As this table Indicates, annual earnings has been the primary 
outcome measure, which Is very consistent with recent federal 
employment and training legislation. Moreover, Social Security 
Administration records have been the main source of earnings data. 
Although SSA earnings records have several advantages (e.g., they are a 
cost-effective source of longitudinal data that are measured comparably 
for participants and comparison group members), they have several 
potential disadvantages Including coverage problems, exclusion of 
earnings beyond the taxable maximum, and considerable delays In 
obtaining reliable data (of up to three to four years). Moreover, when 
SSA earnings Is the only outcome measure available, the evaluation is 
1 Imited to estimating impacts on an annual basis and, except for 
examining the probability of working at all during a year (indicated by 
positive SSA earnings), It is not possible to examine the effects of 
employment and training programs on the components of earnings (e.g., 
hourly wage rate, hours worked per week, weeks worked per year). As 
the first column in Table 1 indicates, the only previous studies that 
were able to examine the Impact of employment and training programs on 
the components of earnings were those for which relatively expensive 
primary data collection efforts were undertaken. Moreover, Information 
on short-term earnings Impacts—those within approximately three to six 
montos after termination—can only be provided through primary data 
collection efforts or through the use of UI earnings records. 

The second column of Table 1 indicates the subgroups of participants 
for whom separate Impact models were estimated. These studies suggest 
that net Impact models should be developed separately for men and 
women. This Is in part because the relationship between earnings and 
various socioeconomic and demographic characteristics is different 
between men and women and also because of the considerable evidence 
indicating that employment and training programs result in sizable 
earnings gains for women (particularly nonblack women) , whereas the 
Impact on earnings for men Is consistently less, and 1n several 
Instances has been estimated to be negative. 

Although the evaluations of MDTA programs tended to estimate separate 
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Table 1 



SUMMARY OF SELECTED FEATURES OF PREVIOUS IHPACT EVALUATIONS 



titer 
19731 



, HcGuIre, and 
tt (1979) 

(1979) 



(19781 



d Boms 



er (1978) 
5 (1981) 
et al. 



and 

dorfer (1970) 



Brennan, and 
(1970) 



ner Assoc. 



Outcome Heasurc(s) 
SSA earnings 



SSA earnings and whether 
employed (SSA greater than }50) 

Interview-reported 
annual earnings 



Interview-reported weekly 
earnings and employment status 

SSA earnings 



Interview-reported annual 
earnings and welfare grants 



Interview-reported earnings, 
employment, hours worked, wage 
rate, welfare grant 

Interview-reported earnings, 
employment, hours worked, 
hourly wage, schooling, 
criminal activity 

Interview-reported wage 
rates, earnings and employment 
status 



Annual UI earnings 



Interview-reported earnings 
and wage rates 



Subgroups 

White males 
Black males 
White females 
Black females 

Hales 
Females 

Separate major 
program types 



White males 
Black males 

White males 
Black males 
White females 
Black females 
(separately for each 
major program type) 

Hales 
Females 



None 



Hales 

Females without 

children 
Females with children 

White males 
Black males 
White Females 
Black Females 

None 



None 



Participant Group(s) 

Jan. 1964 - Karch 1964 
HOT A classroom training 
enrol lees 



1969 - 1971 MOTA classroom 
training and OJT enrollees 

1968 - 1970 MOTA classroom 
training, J08S, Job Corps, 
and NYC enrollees 

1968 - 1970 HOTA classroom 
training enrollees 

1968 - 1970 MOTA classroom 
training, JOBS, Job Corps, 
and NYC enrollees 



Spring 1974 WIN enrollees 



Supported Work 
long-term AFDC recipients 



April 1977 Job Corps participants 



1965 .. 1967 NYC/IS and NYC/OS 
enrol lets 



1966 NYC/OS termlnees 
3n Indiana 



CEP participants 



Treatment Measures 
Participant dummy 

Participant dummy 



Participant dummy for 
separate program models 



Weeks participated 
(and weeks-squared) 

Participant dummy and 
weeks participated 
Interacted with all 
socioeconomic 

characteristics for separate 
program model s 

Participant dummy (sometimes 
interacted with services 
received) 

Participant dummy 



Participant dummy 



Participant dummy 



Participant dummy (sometimes 
Interacted with hours In 
program and sex) 

Hone 
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Table 1 (cont.) 

SUMMARY OF SELECTED FEATURES OF PREVIOUS IMPACT EVALUATIONS 



Outcome Heasure(s) 
) SSA earnings 



ohnson, SSA earnings 
84a) 



1984) SSA earnings 



(1964) SSA earnings 



1982) 



SSA earnings 



SSA earnings 



Subgroups 
None 



Adult males 
Adult females 
Young males 
Young females 
(sometimes separately 
for each major program 
type) 

Adult minority males 
Adult white females 
Adult minority females 

Youths: 

White males 
Black males 
Hispanic males 
White females 
Black females 
Hispanic females 

Economically 
Disadvantaged: 
White males 
Minority males 
White females 
Minority females 

Adult males 
Adult females 



Adult males 
Adult females 



Participant Group(s) 

FY 1976 CETA enrol lees who 
stayed at least 8 days and 
terminated by Dec. 31, 1976 



Calendar year 1976 CETA enroll ees 
who terminated by Dec. 31, 1977 



FY 1976 CETA enrollees who were 
at least age 23 



FY 1977 CETA youth enrollees (under 
age 23), and economically disadvantaged 
and welfare recipient enrollees 



Jin. 1975 - June 1976 CETA enrollees 
who were it least age 25 and stayed 
it least 8 days 

FY 1976 CETA enroll ees who were 
it least ate 22, stayed at 
least 8 days, and terminated by 
Dec. 31, 1976 



Treatment Measures 

Participant dummy 
(sometimes Interacted with 
program activity dummies or 
participant characteristics) 

Participant dummy 
(sometimes Interacted with 
participant characteristics) 



Participant dummy 

(sometimes interacted 

with program activity dummies) 

Participant dummy 
(sometimes interacted 
with program activity 
dummies) 



Participant dummy 

(sometimes interacted 

with program activity dummies) 

?rc«rim activity dummies 
Interacted with race 
dummies, age, end a 
quadratic In lonetn of 
stay In days 
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tlLSPii "?? el * f0r "H* 65 and blacks » more recent evaluation have 
£ f \V$l mte < "PS*! m ° delS by aSe 9rou P s - In fact » because of 
Mcult1es 1 1 volved 1n developing valid net Impact estimates for 
youths, many recent studies have estimated models only for adult men 
and women. The problems for youths are primarily (l) that earnings Is 
ULS i ♦y pr0P i M A te outcome nieasure for Individuals who may return to 
school -the relative mix of schooling, market work, nonmarket work, and 
I?«? re ,f h ? n5es ra P 1dl y over time for youths— and (2) that it is v*r* 
?Jifl2 A re71able ' Batch « , comparison group for youths with 

limited and highly variable earnings histories. Moreover, a 
considerable amount of pre- and post-program data on schooling behavior 
are required to obtain valid net Impact estimates, much more data than 
are generally available on existing data sets. As such. It may not be 
feasible to develop a state-level JTPA net Impact model for youths that 
provides valid results and can be Implemented 1n a cost-effective 
manner. 

Finally, In last two columns of Table 1 we summarize the 

participant groups chosen for analysis and the variables Included In 
the model to measure the treatment effects. For the most part, these 
studies focused on estimating the average impact of program 
participation on earnings for the selected subgroups. Because li. many 
cases the subgroups of Interest were specific program activities, this 
resulted 1r> numerous net Impact estimates by program (treatment) type. 
The only other dimension of the treatment that was examined In a few of 
these studies Is length of program participation. The results from 
these studies Indicate that net Impacts vary by program activity and 
length of stay. Although fewer programs are generally offered under 
JTPA (as compared with CETA), and the average length of stay In JTPA Is 
much less than In CETA, It will still be Important to develop models 
that examine these potential differences. 
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CHAPTER 3. CONCEPTUAL FRAMEWORK 



An Important element 1n the design of a state-level JTPA net impact 
model 1s the conceptual framework. The conceptual framework defines 
the scope and focus of the analysis and guides the research design and 
analysis methods. In particular, the conceptual framework Identifies 
the key research questions to be addressed, the outcomes to be examined 
based on those questions, the participant groups and program activities 
(treatments) to be Included to address the questions of Interest, and 
the specific definitions of the outcomes, treatments, and variables 
that affect the relationship among treatments and outcomes. In this 
chapter we describe a conceptual framework for conducting a state-level 
JTPA net Impact analysis. 

In describing the various components of the conceptual framework, we 
first discuss the alternatives considered and then Indicate our 
recommendation. Each recommendation 1s 1n part based on the results of 
the literature review described 1n Chapter 2. This enables us to focus 
on the major outcome measures, treatments, and participant groups of 
Interest so that key research questions that have been examined 1n the 
literature can be replicated. In addition, our recommendations are 1n 
part based on the recognition of the severe resource constraints that 
states and SDAs face, as well as certain other practical considerations 
such as data availability. The two most Important practical 
considerations that affect the conceptual framework are: (1) states 
and local SDAs will not generally be able to Implement an experimental 
design; and (2) states and local SDAs will not generally be able or 
willing to conduct follow-up Interviews with a large sample of 
participants and comparison group members. Although these 
considerations directly affect the comparison group strategy described 
1n Chapter 4, they also Indirectly serve to limit the scope of the 
conceptual framework as described below. 



GENERAL RESEARCH QUESTIONS 

In specifying the major research questions that an analysis of the net 
Impacts of JTPA programs should address, 1t 1s Important to recognize 
that JTPA may affect different groups 1n different ways and at 
different levels. For example, one could specify the general research 
questions from any one of the following four perspectives: 
participants, employers, the government, or society as a whole. The 
questions of Interest vary considerably among these four groups. 
Society as a whole 1s most concerned with whether JTPA results 1n a net 
Increase 1n the resources available (I.e., Increases output--GNP) 
Government 1s primarily concerned with whether JTPA Increases tax 
revenues and reduces transfer payments. Key employer questions concern 
whether JTPA reduces hiring and training costs, or Increases 
productivity. At the 1nd1v1dual-part1c1pant level, the key research 



questions relate to Improvements 1n an Individual's post-program labor 
market experiences. Because evaluating the Impact of JTPA from the 
social perspective 1s beyond the scope of the state-level model, and 
because employer benefits are being examined separately in Volume 4 in 
this series, 1n this section we discuss the general research questions 
that should be included 1n an analysis of the net Impact of JTPA on 
participants' post-program labor market experiences. 16 Research 
questions that are of concern to state and local governments are 
Included only to the extent that JTPA programs affect the likelihood 
that trainees receive welfare payments. 

Our primary goal 1n developing a state-level JTPA net Impact model 1s 
to determine the extent to which JTPA employment and training programs 
Improve the labor market experiences of participants relative to what 
their experiences would have been 1n the absence of the program, that 
1s, relative to a comparison group of otherwise similar 
nonpartldpants. The average net Impact of JTPA programs on 
participants' post-program labor market experiences will give 
policy-makers an Indication of the overall effectiveness of these 
programs that will more than meet a state's Increased accountability 
responsibilities under JTPA. Moreover, since most previous analyses 
primarily focused on estimating average Impacts for the nation as a 
whole, this will provide a benchmark for comparison purposes. 

In order to maximize the amount of policy-relevant Information, 
however, 1t 1s Important that the analysis be designed 1n such a way 
that 1t gives policy-makers a disaggregated view of how the program 
works and the effects 1t has on different types of participants. That 
1s, although 1t 1s Important to know whether the mix of JTPA programs 
1s effective on average, for policy purposes 1t 1s perhaps more 
Important to probe beneath the average Impacts to Identify the relative 
effects on different subgroups. Moreover, because of the many changes 
from CETA to JTPA (both 1n terms of program services provided and 
characteristics of participants served), 1t 1s Important to Identify 
the composition of net program Impacts 1n order to determine the 
comparability of the findings with previous studies. Thus, as we 
describe below, the net Impact model will be designed to address 
research questions concerning two key dimensions: (1) whether certain 
participant groups benefit more from JTPA than other subgroups, and (2) 
whether the net Impact of JTPA differs among program activities 
(treatments) for the program as a whole and for certain demographic 
groups. 



It should be noted that positive net Impacts on participants' 
labor market experiences 1s a necessary condition for the program to be 
considered successful from the point of view of government or society 
as a whole. That 1s, only 1f JTPA Increases participants' earnings, 
and there are no off-setting displacement effects (1.e. r the earnings 
of nonpartldpants are not reduced), can government obtain additional 
tax revenues or can society as a whole have access to additional goods 
and services. Thus, our focus on the net Impact of JTPA on 
participants' post-program outcomes can be viewed as the first step 1n 
a more comprehensive evaluation of JTPA. 
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^rhf r ? 9ram ., impaC K ts wi11 prov1de A"Portant information 9 Zt the 
mechanisms through which the program produces Its effect* it Ji? 
also provide information about V geneKTlzab 11 ty of the results ana 
the comparability of the results relative to previous ItuMet Far 
groSp "'for' ^tSS^J^SL are concentrat^Vnly 

T^? 9 * lT Ct5 * re ver * different from those ^ reporte^Tn nrevl^s 
studies the subgroup analysis may Indicate reasons why For eiamole 

Mo KlnlJgtrlTc^ 3 ^ ^ ™ ^ZlV V 

shed light on whether the average effects of JTPA are homogeneous 
across program activities, whether all program activities result ?n 

cTwrnes'^uir^eT^l ° f + J art,C,pants ' -n"l*1ch W p^r* 
activities result 1n the largest net benefits. Although such analyses 

will provide important information on relative program effectives 

Jat The costs 1 ^ 1 "*. 10 " 5 f ° r targetin9 practices « one must recogn tie 
fn \*1 \ StS ° f c var1oys Program activities differ considerably and 
in the absence of a cost analysis, no conclusions can be made about 
which program activity Is most cost-effective. 

?el a t«S at t e n" le tt e J «•*, 1mP c Ct model wil1 also examine ^y questions 

mptcl? Srv hv iP^h °r " et 1mPaCtS 3nd the extent to wh ' ch ne t 
provide addHioJi ?2 of Program participation, such analyses will 
pVo^uces ffs eff^ctJ °7^ tl0 \r the roechanisro s through which JTPA 
Kecess*™ VJ Hi^fn *« . formation on the timing of net impacts Is 
necessary to distinguish whether (a) the program has only a short-term 
Thlf V, fro V b J * he im P act expected to persist In the long-term 
call J; pa, ; t,cu1a rly important when making Judgments on the Tola te rn 
pTo^aTipn^ The differential analyses by 

MtXlTipI ™ ° f + Sta Jf..? nl help distinguish whether (a) program 
activities generate different Impacts and length of stav is 

?a" nTVhan' tt°Vrtl I 6 " 9 *" ° f Stay ,n any P^'» '» Ant 
be of consIS^hi. Pr ° 9ram activit y- Such information should 

De of considerable use to program operators who have limited 
Information to accurately Judge the outcomes of their programs 

finally, the model will be designed to address Important resparrh 
questions concerning the extent to which net Impact? vary by local 
environmental conditions such as the unemployment rate or urban/ "a 

daL to addr P C s a , USe P K eV 4 i0US nati ° nal Studies did not have the necessary 
data to address such Issues, It may be possible for the state level 

eft ti!e°n e s1 ke r 3 m \*" ""lotion t! what Is known about the 
effectiveness of employment and training programs tha+ ooerate In 
different environmental conditions. operate in 

l^LJZl^l objectives of the state-level net Impact model can be 
summarized by a series of key research questions to be addressed: 
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• What Is the overall net Impact of JTPA programs on participants' 
post-program labor market experiences? 

• How do the net Impacts change over time? 

• For which program activities (treatments) are the net Impacts 
the largest? That Is, which program activities result In the 
largest net benefits to participants? 

• For which groups of participants are the net Impacts the 
largest? That Is, which subgroups gain the most from 
participating In JTPA? 

• Do Individuals who remain In JTPA longer experience greater net 
gains 1n labor market outcomes? 

• How does the net Impact of JTPA vary by local program 
environmental conditions? 

These are the general research questions that the state-level JTPA net 
Impact model will examine. Below we begin to make these questions more 
specific by Indicating the outcomes that will be used to measure 
participants' labor market experiences, by Indicating the participant 
subgroups to be examined, by Identifying the types of program 
activities that will be examined, and by Identifying the local program 
environmental conditions of Interest. 



PARTICIPANT OUTCOMES 

In selectlny the appropriate participant outcomes to be examined In a 
net Impact analysis of JTPA, one should begin with the goals set forth 
1n the legislation. As Indicated earlier, Section 106 of JTPA 
explicitly states that "Job training Is an Investment In human 
capital," and that the "return on the Investment" for adult training 
programs funded under Title II-A Is to be measured by the "Increase In 
employment and earnings and the reduction In welfare dependency 
resulting from participation In the program." The Act also provides 
examples of Indicators of these basic outcome measures that Include 
placement In unsubsldlzed employment, Job retention, and hourly wage 
rates. In addition to the three outcomes listed for adult programs, 
the Act also specifies other outcomes for youths aged 16-21, 
Including: attainment of employment competencies recognized by local 
private Industry councils (PICs); completion of a major level of 
schooling or the equivalent (e.g., elementary, secondary, or 
postsecondary); and enrollment In other training or apprenticeship 
programs or enlistment In the Armed Forces. 

In determining the general outcome measures that are to be used to 
represent the changes In the labor market experiences of participants 
due to JTPA, It Is Important to keep several factors In mind. First, 
one must ensure that the outcome measures used are consistent with the 
objectives stated In the legislation. Second, to the extent possible, 
the outcome measures should be consistent with those examined In 
previous studies so that key research questions can be replicated. 
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2 ^nihi? VnH ""A 1n 3 1121 1mpact anal y s1 s, the outcomes must 

arouD nlllr, "^ably measured for participants and comparison 
group members. Finally, the outcomes selected must Ticognlze the 
11m ted resources available to states and local SDAs for conducting 
evaluation activities Taken together, the latter two factors suggest 
possible Ce rely1n9 on a 9 encv administrative data to the extent 

Based on these considerations, we recommend that the general 
Include- outcome meas "res In a state-level JTPA net Impact model 

• Whether employed; 

• Earnings; 

• Whether receiving welfare grants; and 

• Welfare grants received. 
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These outcome measures are consistent with the major objectives of the 
legislation and for the most part greatly exceed the outcomes examined 
In previous national studies of the net Impact of employment and 
training programs. Moreover, as we describe below, comparable data on 
these measures can be obtained for participants and comparison group 
members from state administrative records, a 7o~st-ef fectl ve data 
source. Although potentially useful Information on the mechanisms 
through which employment and training programs Increase earnings could 
be provided by examining other outcome measures such as employment 
intensity (e.g., hours worked per week, weeks worked per year), hourly 
wage rates, Job retention, or the additional outcomes listed In the 
legislation for youths, such outcome measures require the collection of 
survey data from both participants and comparison group members, which 
would greatly Increase the resources required to Implement the net 

JEX. S?h \aa?£ th , US reduce 1ts P rac ^"l usefulness. 17 However, 
states with additional resources, or particular Interest In some of 
these other dimensions of labor market experiences, are encouraged to 
collect the necessary survey data and follow the research design and 
analysis plans described In subsequent chapters to estimate the net 
impacts of JTPA on these additional outcomes. 

17 An additional outcome measure that could also be created from 
state administrative records 1s receipt of Unemployment Insurance (UI) 
payments. Although this outcome measure may be very appropriate for 
III f, r T an, ! , . We excluded 1t from the core set of outcomes for 

the Title n-A net Impact models because: (1) 1t was not Included In 
the legislation as a key goal; (2) 1t was not used as an outcome 
measure 1n any of the employment and training net Impact studies 
described In Chapter 2; (3) any potential Impact 1s likely to be small 
because only approximately ten percent of JTPA applicants are UI 
claimants; and (4) the development of measures of UI payments received 
over time generally requires the accessing of the payment-history file 
which can be a very expensive process. 
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As Indicated above, In constructing the core outcome measures we 
recommend that available program administrative records be used 
whenever possible. In particular, we recommend that the earnings and 
employment measures be constructed from State Unemployment Insurance 
Wage Records and that measures of welfare grants received and welfare 
dependency status be created from State Welfare Administrative Grant 
Records, Below we describe the advantages of these data sources, their 
limitations, and the specific outcome measures to be created. 



Earnings and Employment Outcome Measures 

State UI Wage Records are an excellent source for developing measures 
of earnings.! 8 In fact, they have several advantages as compared to 
other administrative data sources and to survey data. For example, 
unlike SSA earnings records, that have been used In most previous 
studies, all earnings In covered employment are reported In the UI wage 
records (I.e., earnings are not "top-coded 11 at the Social Security 
taxable maximum), and data are available on a relatively timely basis 
(usually within three months of the end of the quarter of Interest). 
Moreover, unlike SSA data that are only available on an annual basis, 
UI wage records are available on a quarterly basis. This enables one 
to construct both short- and long-term outcome measures to better 
examine the timing of JTPA Impacts. Finally, although survey data 
allow for much more flexibility and a greater range of outcomes, UI 
wage records are not subject to Interviewer biases, to problems that 
arise from some respondents reporting net (after-tax) earnings and 
others reporting gross (before-tax) earnings and, for the most part, 
they are not affected by response-rate problems that can plague survey 
data collection efforts. 

The primary disadvantage of using State UI Wage Records for developing 
earnings measures Is that these data are not available in all states. 
For example, in 38 states and the District of Columbia, employers are 
required to report the quarterly earnings of all employees covered by 
State Unemployment Insurance laws. These states are known as 
"wage-reporting states." The remaining 12 states are known as 
"wage-requesting states," and they request wage Information from 
employers only for Individuals who file a UI claim. The current 11st 
of wage-reporting and wage-requesting states 1s provided 1n Table 2. 
Although one could not currently construct meaningful earnings measures 
for all participants and comparison group members from UI records 1n 
wage-requesting states (since they are only available for Individuals 
who apply for UI— a very nonrepresentatl ve sample), the Deficit 
Reduction Act of 1984 requires that all states effectively become 
wage-reporting states by 1988. Thus, by 1988 all states should be able 



Although State UI Wage Records are an excellent source for 
developing measures of earnings, they are generally not a very good 
source for developing measures of employment Intensity. In almost all 
states, the only employment measure that can be developed 1s whether 
the Individual worked 1n a given quarter. The only exception 1s 
Washington State, which also has Information on the number of hours 
worked 1n the quarter. 
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Table 2 



CURRENT LISTING OF WAGE-REPORTING AND WAGE-REQUESTING STATES 



Wage-Reporting States 



Wage-Requesting States 



Alabama 

Alaska 

Arizona 

Arkansas 

California 

Colorado 

Connecticut 

Delaware 

Florida 

Georgia 

Idaho 

Illinois 

Indiana 

Iowa 

Kansas 

Kentucky 

Louisiana 

Maine 

Maryland 



Mississippi 
Missouri 
Montana 
Nevada 

New Hampshire 
New Mexico 
North Carolina 
North Dakota 
Oklahoma 
Oregon 

Pennsylvania 
South Carolina 
South Dakota 
Tennessee 
Texas 
Virginia 
Washington 
West Virginia 
Wyoming 



Hawaii 

Massachusetts 

Michigan 

Minnesota 

Nebraska 

New Jersey 

New York 

Ohio 

Rhode Island 
Utah 
Vermont 
Wisconsin 
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to use the net Impact model to examine earnings gains using UI Wage 
Records. Moreover, some wage-requesting states (e.g., New York) could 
use the model at this time by accessing comparable earnings records 
maintained by the Department of Revenue, provided the necessary 
Interagency agreements could be worked out. 

Other disadvantages of using State UI Wage Records for developing 
earnings measures Include potential nonreportlng problems, problems 
caused by Individuals living near state borders, and differences 1n 
state practices for collecting and retaining these data. The 
nonreportlng problems (e.g., wages of employees of federal, state, or 
local governments and self-employed Individuals are generally not 
reported), are not major: approximately 80 percent of all state wages 
are Included In State UI Wage Records. Thus, earnings measures 
constructed from these data should be comparable among participants and 
comparison group members who work within a given state. Because the 
system 1s state-based, however, It Is Impossible to distinguish 
Individuals who work across the border In a different state from 
Individuals who do not work In covered employment. Thus, unless 
Inter-state agreements can be worked out to access UI Wage Records, 
there may be some problems In estimating the net Impacts of JTPA on 
earnings for large SDAs located near state borders. 19 

The final Issue, retention of UI Wage Records, Is particularly 
Important because detailed pre-program earnings Information Is critical 
for obtaining valid estimates of the net Impacts of JTPA on 
post-program earnings. Because the wage records are created for 
determining UI eligibility and benefit level, they are actively 
maintained for at least the "base period" used to determine UI 
eligibility and benefits In the state, which generally varies from four 
to six quarters. In many states (Including the State of Washington), 
however, these wage data are actively maintained for at least three 
years. Although procedures can be designed to access adequate 
pre-program earnings Information for analysis even 1n states that have 
only four quarters of data available at any one time, these procedures 
are operationally awkward and potentially burdensome. Instead, states 
that are seriously Interested In conducting a net Impact analysis might 
consider adjusting their archiving and retrieval practices so that at 
least three years of UI wage records are easily accessible. 

Using State UI Wage Records, erasures of pre- and post-program 
employment status and earnings will be developed for both participants 
and comparison group members. In selecting the specific periods of 
measurement for the earnings and employment outcom variables, we were 
guided by several factors. First, the periods of measurement should be 
consistent with those examined 1n previous studies so that key research 
questions can be replicated. Because, « flescrfbed 1n Chapter 2, most 
previous studies examined net Impacts on earnings approximately one to 



It should be noted, however, that because net Impact estimates 
are based on differences 1n earnings between participants and 
comparison groups members, as long as JTPA does not affect the 
probability of moving out of state or working across state borders, 
this problem should not bias the results. 
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^i ¥ea ^L?O er f^J™ te ? ttnat,on ' n w»s felt that the state-level 
moder should Include outcome measures at least a year after 
timtiwfcton. Second, to examine the timing of Impacts, some short-term 
and intermediate measures must be developed. Finally, 1n selecting the 
specific measurement periods. It must be recognized that the Ul wage 
records are available only on a quarterly basis. 

Based on these considerations, we recommend that the state-level JTPA 
net impact model examine the following earnings and employment outcome 
measures using UI Mage Records: 

• Quarterly, semi-annual, and annual earnings; and 

• Quarterly, semi-annual, and annual employment status (i.e., 
whether employed— based on whether UI Wage Records are positive 
in the particular period). 



These outcome measures capture the range of short-term and relatively 
long-term Impacts that could be observed within a two-year program 
analysis cycle and will provide valuable Information on the benefits to 
participants from JTPA. States that are interested in additional 
information on the duration and timing of program benefits should 
consider including annual earnings 1n a second post-program year as 
another outcome measure. 



Welfare Outcome Meaatsrae 

The other major outcome measures to be examined relate to welfare 
dependency He recommend that these outcome measures be developed from 
State Welfare Administrative Payment Records, which appears to be the 
only cost-effective source of welfare data for both participants and 
comparison group members. However, unlike the state UI Wage Records 
that are quite consistent across the wage- reporting states, there is 
considerable variation 1n state and local welfare administration and 
record-keeping practices, as well as differences in the degree of data 
automation and retrieval capabilities. As such, It will clearly not be 
feasible for all, or perhaps even for most states to Implement this 
component of the net impact model. Nevertheless, we believe that 
states with the capability and Interest can learn much about the net 
impact of JTPA programs on welfare dependency by Implementing this 
component of the model. Below we briefly discuss some of the Issues 
involved 1n accessing and using welfare administrative records to 
construct reliable measures of welfare dependency and indicate the 
recommended outcome measures. 

In developing operational definitions of outcomes to measure the 
reduction 1n welfare dependency due to JTPA, it 1s Important to keep in 
mind the focus of the legislation. JTPA explicitly refers to measures 
of reductions of the number of individuals and families receiving cash 
welfare payments as well as the amounts of these payments. Because 
there are several public assistance programs that provide cash welfare 
payments, one is Immediately confronted with the Issue as to which 
welfare programs to include in the analysis. The primary cash 
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assistance programs Include Aid to Families with Dependent Children 
(AFDC), General Assistance (GA), and Supplemental Security Income 
(SSI). It 1s Important to recognize that these programs are 
administered at various levels of government and have different 
eligibility criteria, payment levels, and potential lengths of 
participation. Because of these differences, the programs often do not 
share a common data base, which leads to additional complications for 
analysis purposes. As a result of these complications, and recognizing 
that reduction 1n AFDC grants 1s the most policy-relevant potential 
outcome of JTPA as 1t relates to welfare dependency Issues, we 
recommend that the welfare dependency outcome measures only be 
concerned with the AFDC program. 

It must be recognized, however, that even after limiting the scope rf 
the analysis to AFDC grants, 1t will generally be possible to Implement 
this component of the net Impact model only 1n states that administer 
the AFDC program (I.e., not 1n states 1n which the program 1s 
county-administered) and that have sufficiently sophisticated data 
systems so that necessary pre- and post-program AFDC grant records can 
be retrieved. Moreover, even 1n state-administered AFDC programs with 
automated grant records available at the state level, several 
additional complications must be dealt with. First, although the 
program data system should contain the actual AFDC grant to recipients 
for the current month, grant amounts for earlier months may be very 
difficult to obtain 1n some states and will generally require 
searching a large historical file. The focus of the data system on 
recording current grants to AFDC recipients 1s easily understood 
because of the program's emphasis on providing assistance to the 
current case load. For analyzing the net Impact of JTPA on AFDC 
participation, however, historical grant records (over at least two to 
three years) are necessary to control for potential differences between 
participants and comparison group members on pre-program AFDC 
participation. As such, the model cannot be reliably Implemented for 
AFDC programs without detailed historical Information unless reasonably 
expensive personal surveys are administered to obtain both pre- and 
post-program AFDC grants for participants and comparison group members. 

A second potential complication Involves the ability to Identify the 
AFDC grants received by specific JTPA participants and comparison group 
members. In most states, a welfare case number 1s assigned to Identify 
the "assistance unit," and the history file contains grants to that 
assistance unit over time. Problems arise, however, because not all 
state AFDC data systems have the Social Security number (SSN) for every 
member of the assistance unit. That 1s, because the S3N 1s the only 
link between an Individual 1n the analysis sample and his/her welfare 
data, 1t 1s very difficult to determine whether a specific Individual 
Is receiving AFDC when the SSNs are not available for all members of 
the assistance unit. Moreover, even 1f 1t 1s possible to determine 
whether a specific person 1s currently 1n the assistance unit, because 
the history file usually corresponds to an assistance unit and does not 
have Information on who 1s 1n the unit over time, It 1s possible that 
the specific Individuals of Interest (I.e., JTPA participants and 
comparison group members) may not have been 1n that unit 1n earlier 
months. As a result, the pre-program AFDC history for the unit may not 
accurately reflect a person's welfare recipiency status during this 
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period. This Is particularly a problem for Individuals who experience 
a marriage or divorce, or who change living arrangements. 

In creating AFDC dependency outcome measures 1t 1s also Important to 
recognize the Implications of the earned Income disregard provision of 
the program. Specifically, during the first four months after an AFDC 
recipient begins working, the "$30 and 1/3 rule" 1s applied. This 
provides that the first $30 of earnings and one-third of the remainder 
wll not reduce the welfare grant. In addition, work expenses (up to a 
maximum of $75) and child-care costs (up to a maximum of $160 per 
chl id) are deducted from earnings before the earnings are counted 
against the welfare grant. Thus, for the first four months, AFDC 
recipients can earn a considerable amount with a relatively modest 
reduction In the grant. After the first four months, however, the 
grant Is reduced dol lar-for-dol lar after work and child-care expenses 
are deducted.^" Because of the four-month cutoff of the earned 
Income disregard provision, and because the AFDC system generally takes 
a month or more to adjust grants to reflect earned Income, one might 
consider adjusting the outcome periods for the AFDC dependency measures 
and make them somewhat longer than for the employment and earnings 
measures. This would be particularly useful 1n states with relatively 
brief JTPA program treatments. However, 1n states with average-to-long 
JTPA treatments (e.g., four months or longer), the earned Income 
disregard should not Introduce any serious analytical complications for 
post-program net Impact analysis. 

Based on these considerations, we recommend that the outcome measures 
related to reductions 1n welfare dependency be as follows: 

• Quarterly, semi-annual, and annual AFDC grants received; and 

• Quarterly, semi-annual, and annual AFDC participation status 
based on whether AFDC grants were received during those periods. 

Because the AFDC grants data are available monthly, and the UI Wage 
Records are only available quarterly, the AFDC measures will be 
aggregated over the appropriate months to correspond to the same 
calendar quarters as the post-program earnings measures developed 
from UI Wage Records. These measures capture the short-term and 
relatively long-term welfare dependency Impacts that could be 
observed within a two-year program analysis cycle and are consistent 
with the objectives of the legislation. It should also be noted 
that the six-month measure 1s consistent with the recommendation of 
a recent study for DOL concerning the measures to be used 1n setting 
post-program performance standards for Title II-A programs for adult 
welfare recipients (Berkeley Planning Associates, 1984). 



It should also be noted, that 1f 
more of the state's standard of need, 
removed from the welfare rolls without 
grant levels. 



the person earns 150 percent or 
she or he will automatically be 
any transition period of reduced 
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PARTICIPANT SUBGROUPS 



The next conceptual framework Issue concerns the participant groups for 
which the outcome measures described above will be examined to 
determine the net Impacts of JTPA programs. Several Issues concerning 
participant groups must be addressed. For example, 1t must be decided 
whether all JTPA participants are to be Included or whether the averagp 
net Impact estimates are to be based on a sample of participants that 
excludes certain types of Individuals. In addition, decisions must be 
made on the participant subgroups for which separate net Impact 
estimates will be derived to Identify the types of Individuals who gain 
most from participating In JTPA. A related Issue concerns the 
subgroups for which entirely separate net Impact models are to be 
estimated. Finally, operational definitions of the participant 
characteristics to be Included as control variables 1n the net Impact 
model must be developed. Because the specific Individual variables to 
be Included In the model must be available and comparably measured for 
both participants and comparison group members, the operational 
definitions of Individual characteristics are constrained by the 
Intersection of common elements In the JTPA HIS and 1n the agency data 
base for the comparison group selected (I.e., assuming surveys of 
comparison group members are not cost-effective). As such, this issue 
depends heavily on the specific comparison group strategy followed, and 
we therefore defer our discussion on variable definitions to Chapter 
4. In the remainder of this section we discuss Issues related to 
participant groups to be Included In the analysis and the subgroups for 
which separate net Impact estimates are to be developed. 

The first Issue concerns whether the net Impact model should be based 
on all JTPA participants or whether certain participant subgroups 
should be excluded. Our recommendation Is to develop a net Impact 
model only for adults. This recommendation Is In part because the 
outcome measures listed above (earnings, employment, and AfDC 
dependency) are not appropriate for youths (particularly In-school 
youths), and no existing data sets Include Information on more 
appropriate outcome measures (e.g., schooling attainment, employment 
competencies) for both participants and comparison group members. In 
addition, our recommendation Is In part due to the extensive evidence 
presented In Chapter 2 documenting the difficulty researchers have had 
In developing a reliable matched comparison group for youths who have 
limited, and highly variable, pre-program earnings histories. Rather 
than design an approach that Is highly likely to yield biased and 
misleading results for youths, we recommend that the net Impact model 
be developed only for adults. 

The omission of youths from the net Impact model Is unfortunate as JTPA 
requires that 40 percent of all funds be expended on this Important 
target group. It must be pointed out, however, that the gross Impact 
evaluation guide (Volume 4) contains more relevant outcome measures 
that can be examined to determine the gross Impacts of JTPA for 
youths. In addition, states that are very Interested In developing net 
Impact estimates for JTPA Title II-A youth programs might consider 
Implementing an experimental design, or alternatively, administering 
relatively expensive Interviews of participants and comparison group 
members to collect the detailed pre-program and post-progr™ employment 



and schooling data necessary for reliable analysis. 



The second Issue concerns whether separate net Impact models need to be 
developed for any specific adult subgroups. As described In Chlpte? 2 

I!^L? r !nl2 U % St + U h d1eS V\Y* tei separate net Impact models for men and 
women, some of these studies also estimated separate models for youths 

T c ' ar l d ot l er . s est1mated separate models by race or ethnicity 
group Separate net Impact models should be estimated for subgroups of 
participants that have different earnings functions, that 1s, that have 
a different relationship between earnings and other demographic 
characteristics. We recommend that separate net Impact models be 
developed for adult men and women because of extensive evidence 
indicating that the relationship between earnings and other demographic 
characteristics 1s very different for these two groups. On the other 
hand, as described 1n Chapter 2, we do not believe 1t 1s necessary to 
estimate entirely separate net Impact models for other subgroups such 
as race or education. This 1s because the earnings functions for these 
subgroups of employment and training participants are generally not 
sufficiently different to warrant the reduction 1n sample size and 
statistical power that would occur by estimating entirely separate 
models, and because the major differences between the groups can 
usually be accounted for through Including appropriate Interaction 
terms 1n the net Impact model. 

Although completely separate net Impact models are not recommended for 
other subgroups, as we describe In Chapter 5 the analysis can be 
designed to examine whether the net Impact of JTPA differs among 
subgroups of Interest. Based on the evidence presented 1n Chapter 2 
It will be Important to Investigate whether the Impact of JTPA varies 
by the following participant characteristics: 



• Age (e.g., less than or equal to 35 as compared to over 35); 

• Ethnicity (whites as compared to blacks and H1span1cs); 

• Educational level (at least a high school graduate as compared 
to others); 

• Marital status (married as compared to unmarried); and 

• Welfare status (welfare recipients as compared to nonredplents) . 



To the extent possible, we recommend that net Impact estimates be 
derived for these subgroups of adult men and women 1n order to 
replicate the research questions examined 1n previous studies and to 
provide valuable Information on targeting Issues. It must be 
recognized, however, that the extent to which these subgroup Impacts 
can be reliably estimated depends on (1) the availability of the data 
Ti uV^x b i Participants and comparison group members; (2) the 
availability of sufficient sample sizes to generate reasonably precise 
net Impact estimates; and (3) the availability of adequate state 
resources to support the additional data processing and analysis 
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PROGRAM ACTIVITIES 



Another Important element of the conceptual framework Is the 
determination of the key program activities (treatments) to be examined 
and the development of consistent definitions of the treatment 
variables. Such decisions have Important Implications for the types of 
analyses that can be conducted to examine relative program 
effectiveness that may 1n turn provide evidence on targeting 
practices. In this section we discuss the major training activities 
being offered by JTPA and provide general guidelines for developing 
operational definitions for the specific program activity variables. 

Section 204 of the Job Training Partnership Act authorizes the 
expenditure of Title II-A funds for over 20 types of employment and 
training program activities. Although the list of potential program 
activities is quite extensive and allows for a large combination of 
potential services, the major program activities provided under JTPA 
include classroom training (CT), on-the-job training (OJT), and job 
^^ ch T ,i s , s1stance < JSA >- I" fact, nearly 90 percent of adult FY 
1984 Title II-A enrollees participated In one of these programs 
Although work experience was used extensively under CETA, only 3 
percent of the adults In FY 1984 were assigned to work experience 
programs. A brief description of each of these program services Is 
provided below: 

• Classroom training involves basic or remedial educational 

training, or occupational skills training to ensure that 
Individuals acquire the ability and knowledge necessary to 
perform a specific job for which there Is a demand. Such 
programs are usually provided 1n a classroom or Institutional 
setting. 

* On-the-job training emphasizes the development of occupational 
skills In an actual work setting, normally In the private 
sector. The programs are designed for participants who have 
been first hired by the employer, and the training occurs while 
the participant is engaged In productive work that provides 
knowledge or skills essential to the full and adequate 
performance of the job. 



It she j Id bo noted that JTPA has considerably altered the mix of 
program activities relative to CETA. In particular, the elimination of 
public service employment programs, and the 15 percent cap placed on 
support services (with a large portion of work experience expenditures 
being regarded as support services), and the Introduction of 
performance standards that emphasize Immediate placement and low costs 
have resulted in major changes In the mix of program activities For 
example, In FY 1982, 13 percent of all CETA participants enrolled 1 5. 
OJT programs and 30 percent enrolled in work experience programs In 
FY 1984, however, 22 percent of the Individuals In JTPA Title II-A 
programs enrolled In OJT and only 7 percent enrolled In work experience 
programs. In addition, job search assistance Is being used much more 
extensively under JTPA. 
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• Job search assistance Includes any training activity that focuses 
on the development or enhancement of employment-seeking skills. 
This service 1s provided to participants who need practical 
experience 1n Identifying and Initiating contact and Interviewing 
with prospective employers. It 1s usually conducted 1n a 
structured setting and can Include approaches such as job-f1nd1ng 
clubs or Instructions for self-directed job-search methods. 

• y°Jj£ experience 1s a short-term or part-time work assignment 

designed to enhance the employab111ty of participants by developing 
good work habits and basic work skills. It 1s primarily Intended 
to assist participants 1n entering or re-entering the labor force. 
Work assignments may be with a public employer or with a private 
non-profit agency; work experience programs are prohibited 1n the 
pr1vate-for-prof1t sector. 



In creating specific treatment variables to represent these program 
activities, a couple of potential problems must be recognized. First, 
although the broad definitions of the major program activities are 
generally accepted, there 1s likely to be considerable variation across 
SDAs In the contents of specific program activities such as length of 
assignment, occupation, or hours per day. This makes 1t particularly 
difficult to create meaningful variables that represent a homogeneous 
treatment. At the same time, however, 1t 1s necessary to do a 
considerable amount of aggregation of activities that are generally 
similar but perhaps far from Identical, because 1t 1s simply not 
possible to reliably estimate the net effects of the virtually 
unlimited number of program activities. Second, not only are there 
differences 1n the degree of treatment within program activity, but 
there are likely to be large differences across SDAs 1n the nature of 
programs provided such that work experience programs 1n a particular 
SDA may be more similar to OJT programs 1n another SDA. Another 
complication concerns the way 1n which the actual training activities 
provided are recorded 1n the program MIS. For example, due to the lack 
of uniform national reporting requirements, some SDAs record 
participation 1n a Job search workshop as job search assistance, while 
others record 1t as classroom training. Such differences 1n the 
content and recording of program activities across SDAs emphasize the 
Importance of conducting a process analysis concurrently with the net 
Impact analysis 1n order to develop meaningful and consistent measures 
of program activities. 

As Indicated above, the ways 1n which the treatment variable* are 
defined will 1n large part be determined by the structure and content 
of the MIS. In addition, they will depend on the specific research 
questions of Interest and the sample sizes of Individuals who 
participate 1n the given program activity. For example, to ensure that 
the treatment variables are as homogeneous as possible, it may be 
desirable to separate classroom training activities that focus on 
remedial education and basic skills from classroom training activities 
that provide specific occupational skills training. At the same time, 
however, 1f the number of Individuals participating 1n each of these 
programs Is too small to produce statistically reliable net Impact 
estimates for the separate activities, 1t may be necessary to collapse 
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these two variables Into one that represents classroom training 
programs In general . 



Thus, although the specific definitions of the treatment variables will 
depend on several factors, we recommend that th* following variables 
should be created for participants for potential Inclusion In some 
models to examine Impacts by program activity and other characteristics 
of the treatment: 

• Participant durwiy variable (1 If JTPA participant; 0 otherwise); 

• classroom training dummy variable (1 1f CT participant; 0 
otherwise) 

Classroom training recrjdlal education 3ummy variable (1 If 
CT program In i emedlal education, Engl Ish as a secc r 
language, or basic skills; 0 otherwise) 

Classroom training Institutional skills dummy variable (1 If 
LT program 1n specific occupational skills; 0 otherwise); 

• On-the-job training dummy variable (1 If OJT participant; 0 
otherwise) ; 

• Job search assistance dummy variable (1 If participated 1n JSA 
job search or placement-related activities; 0 otherwise); 

• Work experience dummy variable (1 If WE participant; 0 
otherwise) ; 

• Occupation of training dummy variables (1 1f In specific l-d1g1t 
DOT code; 0 otherwise); 

• Length of program participation In weeks; 

• Number of hours of training per day; and, 

• Completed training program activity dummy variable. 

Such treatment variables would enable one to replicate all of the 
questions examined 1n previous national studies of employment and 
training programs, as well as several additional questions of Interest. 
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PROGRAM ENVIRONMENTAL CONDITIONS 

The final element of the conceptual framework concerns the program 
environmental conditions to be Included In the net Impact model. By 
program environmental conditions, we are primarily referring to 
characteristics of the labor markets within which the program operates, 
although major SDA characteristics could also be considered. As 
discussed earlier, because of data limitations, previous nations! 
studies have b^en unable to Include any program environmental variables 
1n their models. As such, llttla 1s known about how the net Impact of 
employment and training programs varies by program environmental 
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conditions. At the same time, however, because of the nature of local 
program environmental conditions (I.e., there 1s no with1n-SDA 
variation on these conditions), 1t 1s Important to recognize that It 
will only be possible to obtain reasonably precise estimates of a few 
key conditions, and only 1n states that have a large number of SOAs and 
that exhibit considerable variation In the conditions across SDAs. 
Given these limitations, below we discuss some of the more Important 
comparisons that should be examined In a state-level JTPA net Impact 
model . 

In choosing among the many characteristics of labor markets to Identify 
the few that should be Included In the net Impact model, It 1s 
Important to focus attention on those factors that are most likely to 
affect the employment and earnings experiences of adult men and women. 
Although there are many factors that may affect the employment 
opportunities or demand for an Individual with certain skills, economic 
theory and limited empirical evidence suggest that the most Important 
factors are likely to be (1) the local unemployment rate and (2) 
whether located In an urban or rural area. The first factor has been 
demonstrated to be a key variable In affecting the Immediate 
post-program outcomes of JTPA (West and Dickinson, 1985). Moreover, 
Johnson, Dickinson and West (1985) also provide evidence suggesting 
that the net Impact of the ES Is larger In urban areas and 1n areas 
with lower unemployment rates. Thus, we recommend that readily 
available labor market Information be used to construct measures of the 
local unemployment rate for the appropriate period of analysis so that 
Us effect can be controlled for In the model and so that one can 
estimate separate net Impacts for programs that face different 
unemployment conditions. 

The unemployment rate can be obtained from the Local Area Unemployment 
Statistics (LAUS), published by the U.S. Bureau of Labor Statistics 
(BLS). This Information is available monthly at the state and county 
level and for over 1,000 cities with a population of at least 25,000. 
Aggregate measures of the unemployment rate corresponding to the 
outcome periods of Interest can be calculated as weighted averages of 
the monthly values. In constructing these aggregate measures 1t is 
Important to recognize that the unemployment rate will not generally be 
available for the precise area of Interest. Depending on the size of 
the SDA, the area 1t serves may be either larger or smaller than the 
county or the city for which the Information 1s available. In cases 
where the SDA serves multiple counties, one should calculate the 
appropriate unemployment rate variable by aggregating over the counties 
served by the SDA. For example, one would simply sum the number of 
Individuals unemployed in the various counties served by the SDA and 
divide by the total number of Individuals in the labor force 1n those 
counties during the appropriate months. In cases where the SDA serves 
only part of a given county, one is generally constrained to use the 
county value unless data for the specific cities served are available. 

It may also be possible to provide some Information on how the net 
Impact of JTPA varies by different SDA service delivery strategies. 
The service delivery strategies to be examined should be based on their 
policy Importance to the particular state doing the analysis. 
Moreover, to ensure that the strategies of Interest are dustlnct and 



quantifiable, and that there 1s sufficient variation among SDAs to 
support the analysis, 1t 1s Important that a process analysis be 
conducted. Thus, 1f states with a large number of SDAs (roughly 30 or 
more) are Interested 1n obtaining some Information on how the net 
Impact of JTPA varies by a key service delivery strategy, they should 
first ensure that there are significant differences In this strategy 
across SDAs. Provided It Is possible to quantify these differences, 
one could then use the variable created to determine how the net Impact 
of JTPA varies across SDAs that differ In this strategy . using the 
approach to estimate subgroup effects described In Chapter 5. In 
states with relatively few SDAs, It Is very unlikely that such an 
analysis would provide reasonably precise estimates of the differential 
effects of the strategy of Interest. 
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CHAPTER 4. RESEARCH DESIGN 



To provide valid estimates of the net Impacts of JTPA programs on the 
earnings and AFOC dependency of adult men and women, a research design 
must be developed that contains several elements. First, a sample of 
participants must be chosen so that the results can be generalized to 
the state level and the necessary data can be col lected efficiently, 
with minimum burden to state and SOA staff. Second, a valid comparison 
group must be chosen so that the Impact of JTPA can be distinguished 
from the Impacts of other factors that also affect earnings and welfare 
dependency. Third, the sizes of the participant *nd comparison samples 
must be determined so that program Impacts can be measured with 
precision. Finally, an overall estimation strategy must be developed, 
and analysis models must be specified that can provide valid estimates 
of the net Impacts of JTPA programs on the post-program outcomes of 
participants. In this chapter, we discuss the first three of these 
research design Issues for developing a state-level JTPA net Impact 
model. The overall strategy to estimate net program Impacts and the 
specific models to be estimated are the subject of Chapter 5. 



THE PARTICIPANT SAMPLE 

An Important component of the research design Is the development of the 

participant sample. The two major Issues to resolve In selecting the 

participant sample concern: (1) the Individuals to Include 1n the 

sample frame and (2) the procedure to select participants from the 
sample frame for Inclusion In the analysis. These Issues are discussed 
below. 



The Sample Frame 

The sample frame from which the sample of JTPA participants will be 
drawn 1s an Important determinant of the external validity of the net 
Impact analysis, that 1s, the ability to generalize findings to the 
program as a whole. The sample frame should be representative of all 
JTPA participants so that the analysis results can be generalized to 
the state level , rather than to only particular subpopulatlons of 
participants or Individual SOAs. We considered several potential 
sample frames. Including: 

• All Individuals who are participating 1n JTPA at a given point 
1n time; 

• All JTPA termlnees during a given time Interval; and 

• AIT JTPA enrolleas during a given time Interval. 
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As Indicated In Chapter 2, these are the three major sample frames used 
In previous studies of the net Impact of employment and training 
programs. " Below we discuss the advantages and disadvantages of 
these sample frames as they relate to the external validity of the 
analysis and to the ease of collecting the necessary data. 

£1] — Participan ts at a Given Point In Time . A sample frame that 
contains all JTPA participants at a given point In time has a number of 
disadvantages. The most Important disadvantage of this approach 1s 
that It substantially undersamples Individuals who participate In JTPA 
for only a short period and results 1n a non-representative sample. 
Such an approach also oversamples Individuals who may only participate 
In JTPA during a particular time of the year, e.g., construction 
workers. If program Impacts or the characteristics of participants or 
program activities that are related to JTPA Impacts are In any way 
seasonal, such oversampllng would lead to a seasonality bias. Because 
the average JTPA participant stays In the program for only 
approximately three to four months, this seasonality bias Is 
potentially Important. 

This sample frame also creates data collection problems that make 14 
difficult to draw an appropriate comparison group. This Is because a 
sample of Individuals who are In JTPA at a given time contains some 
Individuals who have just entered JTPA and others who are nearlng 
completion. Because some Individuals are likely to stay In jTPA for up 
to six months or more, such a sample frame would result 1n extending 
the In-program period for the participant group as a whole to a year or 
more. This would make It very difficult to select a comparison group 
that Is similar to JTPA participants on the timing of the pre-program 
decline 1n earnings, which we argued In Chapter 2 Is very Important to 
achieve. In addition, If the long-term stayers were retained 1n the 
sample this would delay the availability of net Impact results. On the 
other hand, excluding such Individuals from the sample would result In 
undersampllng long-term participants, which would reduce the external 
validity of the analysis. Because of these numerous disadvantages, we 
do not recommend that the sample frame be comprised of all Individuals 
who are participating In JTPA at a given point In time, 

A]J Termlnees during a Given Time Interval . The second potential 

sample frame Includes all participants who terminate from JTPA during a 
given time Interval. Unlike the previous approach, because a sample 
frame of termlnees would Include all program completers and dropouts 
during the time Interval selected, such a sample frame 1s 
representative of all JTPA participants and does not lead to 
oversampllng of either long-term or short-term participants. It does, 
however, share a possible seasonality bias with the previous approach, 
unless a one-year Interval Is used to define the sampling period. In 
addition, like the previous approach, because some Individuals remain 
In JTPA for very short periods while others stay for up to six months 
or more, with a sample frame of termlnees It would be very difficult to 




A fourth potential sample frame of participants who enroll and 
terminate during a given time Interval was rejected because It severely 
overrepresents short-term participants. 
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draw a comparison group that 1s similar to participants on the timing 
of the pre-program decline 1n earnings. Moreover, depending on the 
time period selected, and the data retention practices followed, 1t may 
be difficult 1n some states to obtain the necessary pre-program 
earnings and welfare dependency data for long-term participants. 

A sample frame of termlnees has some Important advantages over the 
other two approaches that must be recognized. For example, by using a 
sample of participants that terminated during a particular period 
(e.g., the last quarter of the calendar year), 1t 1s possible to 
estimate post-program Impacts that are measured more closely to the 
point of termination. For example, 1f the outcome period of Interest 
1s three months following the calendar quarter after program 
termination, this period would correspond to from three to six months 
after termination for all Individuals 1f a terml nee-based sample frame 
were used. On the other hand, using an enrollee-based sample and 
assuming that all Individuals terminate from between one to six months 
after enrollment, the first three-month period that would be entirely 
post-program for all enrol lees would vary from between three to eleven 
months after termination for different Individuals depending on program 
length of stay. An additional advantage of a terml nee-based sample 1s 
that 1n states that maintain three to five years of UI Wage Records and 
AFDC Grant Records, 1t would be possible to make just one request for 
the necessary pre-program and post-program outcome measures for a given 
group of termlnees and thus minimize the burden on state data 
processing staff. However, 1n states that maintain only the minimum 
amount of UI Wage Records necessary to calculate benefits (I.e., four 
to five quarters), one would have great difficulty 1n Implementing a 
termlnee-based sample frame. 23 

All Enroll ees during a Riven Time Interval . The final potential 
sample frame—all JTPA enrollees during a given time 1nterval--also has 
the advantage of yielding a representative sample of JTPA participant 
1n which neither short-term nor long-term participants are 
oversampled. Provided the time period selected 1s not too long (e.g., 
a quarter), this approach has the major advantage of allowing one to 
select a comparison group that closely matches participants on the 
timing of the pre-program decline 1n earnings. As discussed above, 
this 1s particularly Important for ensuring valid net Impact 
estimates. In addition, using an enrollee-based sample maximizes the 
amount of pre-program earnings and AFDC data available for the model, 
which Is particularly Important for states that do not retain extensive 
historical data, 



One possible approach would Involve the following steps: first 
draw quarterly samples of program enrollees; then request the 
pre-program UI Wage Records available for those samples at enrollment; 
then ex post construct quarterly samples of termlnees based on 
termination dates; and then obtain post-program outcome data for the 
tersrfnee samples. It should be noted that although such a roundabout 
approach would work 1n the long term (after six months or so), the 
f1*st few quarterly samples of termlnees constructed would greatly 
overrepresent short-term participants. 
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On the other hand, however, this approach shares the problem of 
possible seasonality bias with all procedures unless enrollees during a 
one-year period are Included 1n the sample frame. In addition, as 
Indicated above, because the time required for analysis results to be 
available Increases with the length of the Interval used to define the 
sample frame, such an approach would delay findings more than a 
termlnee sample frame, and result In outcome measures that are further 
away from the point of termination. 

Sample Fr ame Recommendation . As the above discussion Indicates, there 
are advantages and disadvantages to using either an enrol lee-based or a 
terml nee-based sample frame. The particular approach to use depends 
primarily on data availability and on whether one considers It more 
Important to match participants and comparison group members on their 
experiences In the pre-program period, or whether It Is more Important 
to measure outcomes at a common point after termination. Although both 
approaches are viable, based on data availability considerations and 
the Importance of ensuring that the two groups are similar on the 
timing of the pre-program decline In earnings, we recommend that an 
enrol lee-based sample frame of JTPA participants be used for the 
state-level net Impact model . 

In order to precisely define the sample frame to be used In the net 
Impact analysis, two additional decisions must be made. First, If the 
sample frame consists of all participants who enroll In JTPA during a 
given time Interval, that Interval must be specified. As discussed 
above, a long Interval reduces potential seasonality bias problems 
whereas a short Interval can result In more timely evaluation findings 
and also allows one to draw a comparison group that Is matched more 
closely on the pre-program decline In earnings. Because the shortest 
Interval for which the UI and AFDC outcome measures are both available 
Is a three-month period, we recommend that the sample frame consist of 
adult men and women who enroll In JTPA during various calendar quarters. 

The second decision concerns the specific calendar quarters to be used 
for the sampling frame. This decision affects the timing of project 
results, the length of the post-program observation period within a 
two-year program evaluation cycle, and the likelihood of seasonality 
bias. We recommend that the sample frame of participants be comprised 
of adult men and women who enroll In JTPA In each of the four quarters 
of a given program year. Given the delays involved In obtaining the 
outcome measures from agency records (approximately three months) and 
the time required to build the data files, conduct the analysis and 
prepare written reports (approximately six months), we expect that with 
an enrol lee-based samoTe of participants one can obtain net Impact 
estimates for the period one year following the calendar quarter after 
termination for the longest stayers only for the first quarter cohort, 
and that only a three-month net Impact estimate can be obtained for all 
four quarterly cohorts 1n approximately a two to two-and-one-half year 
cycle. Of course, by obtaining additional post-program UI and AFDC 
records for sample members, one could estimate longer-term Impacts, 
although the analysis period would have to be extended even further. 
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Sample Exclusions 



Once the general sample frame 1s chosen, one must then determine 
whether certain types of Individuals should be excluded. Although such 
exclusions reduce the representativeness of the participant sample, 
there may be valid reasons for excluding some participants. For 
example, 1t may be desirable to exclude cases because they lack data on 
critical Items or because they are clearly Inappropriate to Include 1n 
the analysis due to policy or statistical reasons. For the most part, 
previous studies of the Impact of employment and training programs 
discussed In Chapter 2 consistently Imposed an age restriction on 
participants and also excluded Individuals for whom data were missing 
on the outcome measure(s) or any of the key treatment variables such as 
program activity or length of stay. An Issue of Importance for which 
there 1s less consensus concerns whether Individuals with limited 
exposure to the program should be Included 1n the analysis. Belov we 
briefly discuss these potential sample exclusion Issues and offer a 
recommended approach. 

Age. Host previous studies Imposed some restrictions on participant 
age. Although there 1s no universal agreement on the specific age 
range to use, very young participants (e.g., under age 16) have been 
excluded because earnings 1s not an appropriate outcome measure for 
those likely to return to school, and very old participants (e.g., 
those age 65 and older) have been excluded because participation 1n 
employment and training programs among Individuals eligible for 
retirement 1s very rare, and 1t 1s unlikely that a proper comparison 
group could be Identified. Because the state-level net Impact model 1s 
designed to focus on adults only, the participant sample will be 
restricted to Individuals of at least age 22. Because of the 
difficulty of obtaining a valid comparison group for older 
participants, we also recommend that any Individuals age 65 and older 
be excluded from the participant sample frame. 

Hissing Data. Individuals should also be excluded from the sample 
frame 1f they have missing data on key variables. For the most part, 
we do not expect that there will be severe missing data problems with 
the agency records to be used, and most problems can be dealt with 1n 
straightforward ways 1n the analysis. The limited amount of 
missing data Is 1n part a result of the procedures used by many 
agencies to assign "default" values when data are missing. Such 
procedures, however, lead to measurement error, which can also 
Introduce analytical complications as discussed 1n Chapter 5. 



For example, one could substitute the mean value of an 
Independent variable for cases with missing data. Although such a 
procedure preserves the overall mean of the variable, 1t does not 
preserve the variance and results 1n biasing downward the standard 
errors of the estimated coefficients of the variable 1n a regression 
model. A better procedure, which preserves more of the variance of the 
variable, Involves estimating an auxiliary regression equation to 
predict the variable 1n question using cases with com r lete data, and 
then using the predicted value for those with missing data. 
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A more difficult problem arises when Information 1s missing on the 
treatment provided by JTPA. For example, one cannot estimate the 
Impact by program activity or by length of stay for Individuals with no 
designated program activity or for those who have missing data on the 
start and end dates of JTPA participation. We expect there will 
generally be very few problems concerning the omission of program start 
and end dates, 1n part because length of stay 1s necessary for 
adjusting certain performance standards for Title II-A programs. 
However, because there are no uniform reporting requirements for 
program activities, 1t 1s likely that some cases will contain missing 
(or unusable) program activity data. Although Individuals with missing 
data on the JTPA treatment could be Included 1n an Impact model that 
only examined the average effect of JTPA participation, we recommend 
that they be excluded from the sample frame to preserve the statistical 
power of the program activity analysis, provided the reason the data 
are missing 1s unrelated to the program outcome. That 1s, 1f the 
reason the data Items are missing 1s systematically related to the 
Impact of the program, excluding such cases would reduce the Internal 
validity of the analysis. As a result, 1t will be Important to examine 
the missing data problems before making a final decision. 

Multiple Program Activities . In some cases, individuals will be 
assigned to multiple program activities. Provided data are available 
on each of the activities that an Individual participates in, multiple 
activities do not Introduce new problems for the analysis. That is, 
one could Include all cases with multiple activities and directly 
estimate the marginal effects of each program activity using the 
approach described 1n Chapter 5. 25 However, because of data 
limitations, persons who participated in multiple program activities 
caused analytical complications for some previous studies. !"or 
example, the series of recent CETA net Impact studies only had 
available the Initial program activity and whether the individual 
participated 1n multiple activities, but had no information on the 
number or types of other activities. 

In response to this limitation, some of these studies excluded cases 
that participated in multiple activities. However, such an approach 1s 
not recommended because 1t reduces the validity of the overall 
analysis. Other studies Included participants with multiple activities 
and considered them as a separate treatment group or used their Initial 
program activity to represent the treatment. Should a state JTPA HIS 
contain similar limitations, Information would be needed on 



It must be recognized, however, that such analysis would be 
subject to potential selection bias as discussed 1n Chapter 5. That 
is, Individuals who receive multiple program activities may be 
systematically different from those who only receive a single program 
treatment, and these differences 1n personal characteristics may be 
responsible for the observed treatment effect. For example, 
individuals who are less motivated could receive multiple activities 
because the Initial activity was not effective. As a result, a small 
(or negative) estimated net impact for those who participated 1n 
multiple activities would in part reflect the lack of motivation and 
would not measure the true effects of the program activities. 
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the extent to which Individuals receive multiple activities and the 
type of Initial activity to make an Informed judgment on which 
procedure to follow. For example, if a very small proportion of the 
sample were assigned to multiple activities, then 1t would not be 
possible to precisely estimate the Impact for this group. In this 
case, if most of the Initial program assignments Involved relatively 
lengthy treatments such as OJT or classroom training, then the use of 
Initial program activity for those with multiple activities would seem 
to be a reasonable compromise. 

Limited Program Par ticipation . The final issue concerns whether to 
exclude Individuals who participate 1n JTPA for a very limited period. 
As described 1n Chapter 2, some studies chose to Include all employment 
am', tracing participants In the sample, while others Imposed somewhat 
arbitrary restrictions that resulted in excluding Individuals who 
participated 1n the program for only a short period (e.g., a week). 
Although JTPA clearly cannot have a large Impact on those Individuals 
wf.o participate only a few days, exclusion of such Individuals may 
Introduce a selection bias Into the analysis because short-term 
participants are likely to differ from other participants on unmeasured 
characteristics, such as motivation and attitude toward work. For 
example, 1f Individuals leave JTPA early because they found a job, a 
negative bias 1n assessing JTPA Impacts may result because those 
participants who would do relatively well on their own are excluded 
from the participant sample. Alternatively, short-term participants 
might consist of Individuals who would do less well on their own t.ian 
other JTPA participants, and hence their exclusion would result 1n a 
positive bias In the estimated Impact. Both of these selection biases 
threaten the Internal validity of the analysis. We believe It 1s 
preferable to keep the sample of JTPA participants as representative as 
possible and not exclude cases based on length of stay In the program. 
It 1s then possible to examine whether, and 1n what ways, short-term 
participants differ from long-term participants. This will help to 
determine how much confidence to attach to net Impact estimates by 
length of program participation. 



Selecting the Participant Sample 

Once the participant sample frame has been determined, the next step 
Involves the procedures to use 1n selecting JTPA enrollees for 
Inclusion 1n the analysis. In many states, this step will be trivial, 
as all enrollees 1n a given program year will be necessary to provide 
reasonably precise estimates of the average effect of JTPA programs. 
This will also be true for med1um-s1ze states that are Interested 1n 
obtaining reliable net Impact estimates for various subgroups and that 
have the resources necessary to support the analysis. The Issue of 
sampling primarily arises 1n states that serve large numbers of JTPA 
participants. Below we discuss alternative procedures for selecting a 
sample of JTPA participants from the sample frame described above to 
assist states that would find sampling desirable. 

The principal methods that could be used for selecting a participant 
sample Include (1) random selection, (2) stratified random selection, 
and (3) clustering. Because the primary advantage of a clustered 
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sample 1s to reduce the costs of data collection when relatively 
expensive In-person surveys are used, a clustered sample 1s not 
appropriate for the net Impact model that relies on agency data. 
Therefore, below we discuss the relative advantages of selecting a 
random sample versus a stratified random sample of JTPA termlnees. 

Simple Random Sample . Choosing a random sample from the sampling 
frame has the advantage of providing a representative sample of all 
JTPA participants. As a result, estimates of the net Impact of JTPA 
and of the differential Impacts by program activity can be obtained In 
a straightforward manner; no weighting of the estimated Impacts Is 
necessary. In addition, an analysis of the types of Individuals who 
are assigned to various JTPA program activities (I.e., targeting) Is 
straightforward because the probability of selection Into the analysis 
sample Is not related to the assignment to specific program 
activities. The only potential disadvantage to simple random sampling 
1s that 1t Is possible that the resulting sample may be Inadequate to 
estimate the Impact for certain subgroups. For example, the Impacts of 
program activities that occur rarely may have low power even 1f the 
total sample size Is reasonably large. 

Sample Stratification . Sample stratification can potentially Increase 
the statistical power of the net Impact analysis of JTPA programs. 
Depending on the specific research questions of Interest, one could 
consider stratifying on the basis of participant characteristics, 
program activity, or even by SDA, In general, stratification is 
desirable only when the research questions of Interest relate to 
subgroups that occur rarely or that occur so frequently that their 
nonoccurrence is rare. This Is because by altering the design to 
Increase the efficiency of the estimated effects of those subgroups 
that occur rarely, stratification necessarily reduces the power of 
estimates of the overall Impact of JTPA and also complicates various 
analyses because the probability that an Individual Is included 1n the 
analysis sample 1s affected by the stratification. Thus, whether a 
state ultimately decides to stratify, and the particular strata chosen, 
will depend on the research questions of Interest and the distribution 
of the character .tics of participants served and program activities 
provided. 

Sample Selection Recommendation . We recommend that the quarterly 
samples of JTPA participants be selected on a random basis from the 
groups of adult men and women enrollees Included in the sample frame. 
Although adult men and women are served In approximately equal numbers 
by JTPA overall (I.e., 52.8 percent of all Title II-A adult 
participants were women 1n PY 1984), because this varies considerably 
across SDAs, we believe it would be prudent to first stratify the 
participant sample by sex before the analysis samples are selected, or 
else there may be insufficient numbers of either men or women for 
analysis purposes. 26 As described above, because the net impact 
models will be estimated separately by sex, choosing random samples 



For example, 1n some SDAs women comprised as little as 25 percent 
of adult JTPA termlnees in PY 1984, while In other SDAs women were over 
80 percent of all adult termlnees 1n Title II-A programs 1n PY 1984. 
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from the sampling frames, separately by sex, has the major advantage of 
providing representative samples of JTPA adult men and women 
participants, so that the results can be generalized to the state level 
and so that differential Impacts by program activity can be obtained 1n 
a straightforward manner. 

In addition to stratifying the sample by sex, states that want to focus 
their efforts on specific subgroups of adult men or women (e.g., female 
welfare recipients, male or female high school dropouts) may also want 
to consider stratifying the participant sample and oversampllng the 
subgroups of Interest. For example, because of the wide variation 
across states and SDAs 1n the use of work experience programs, states 
that are Interested 1n examining the net Imoact of work experience 
programs would likely need to stratify and oversample participants of 
such programs. Moreover, because job search assistance generally 
constitutes a less Intensive treatment that 1s likely to have a smaller 
average net Impact, a much larger sample of participants 1n JSA would 
be needed to precisely measure the lower expected effect. Thus, for 
states that are very Interested In precisely measuring the marginal 
benefits from JSA participation, such participants would have to be 
oversampled. States that may be Interested 1n stratifying the 
participant sample and oversampllng certain groups should consult a 
statistician or sampling expert to better understand the advantages and 
disadvantages of such an approach and the specific steps to be followed 
1n drawing the sample and conducting the analysis. 

COMPARISON GROUP STRATEGY 

To estimate the net Impact of JTPA on participants' post-program 
outcomes, a method Is needed to gauge what would have happened to 
participants had they not participated 1n JTPA. The standard approach 
for determining the net Impact of a program 1s to compare the 
experiences of persons Influenced by the program (JTPA participants) 
with the experiences of persons who are not Influenced by the program 
(the comparison group). The comparison group 1s used to estimate what 
the experiences of the participants would have been 1n the absence of 
the program. An evaluation that estimates the effects of JTPA without 
reference to a comparison group essentially attributes all gains (or 
losses) to the program, when other factors, such as Improvement 1n 
labor market conditions, might also have contributed to this gain. 
This procedure generally results 1n greatly overstating the benefits of 
program participation per se. To ensure that the differences between 
the experiences of the participant and comparison groups can be 
attributed to the program, the comparison group must have 
characteristics similar tc JTPA participants, and the data available 
must be cosnparab<y measured for the two groups. In this section, we 
describe the advantages 2:10 disadvantages of alternative comparison 
group strategies and present a recommended approach. 

As descr'bvd in Chapter 2, a few recent evaluations have randomly 
assigned applicants to either a treatment group that could participate 
Its the program or to a control group that could not. Such an 
experimental design can potentially allow a valid determination of the 
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program's net Impact because the only systematic difference between the 
treatment and control groups 1s the opportunity to participate 1n the 
program. However, because JTPA is an ongoing program, serious ethical 
and legal concerns preclude the use of a randomly assigned control 
group, particularly for an evaluation that would be conducted on a 
biennial basis. Moreover, to the extent that crossover occurs (I.e., 
control group members reapply and join the treatment group) or that 
substantial attrition occurs, this would threaten the Internal validity 
of the experimental design. Thus, alternative methods that can 
approximate a true control group must be explored. 

One method that has been widely applied 1n other contexts is to use 
each Individual as his or her own control by comparing the Individual's 
pre-program experience with his or her post-program experience. 
Although such comparisons can provide useful Information on gross 
program Impacts, this methodology Is not appropriate for examining the 
net Impact of JTPA for two reasons. First, the key outcomes—earnings 
and AFDC dependency—may change systematically over time, for example, 
as the result of changes In the national economy or changes In state 
welfare eligibility rulis. Pre-program and post-program comparisons 
will confuse the Impact of such changes with the Impact of JTPA. 
Second, it Is very likely that Individuals 1 earnings In the period 
before enrollment In JTPA are atyplcally low and would Increase over 
time even In the absence of participation In the program. Such 
pre-program earnings declines of participants have been extensively 
documented and reflect legislation Instructing program operators to 
serve persons who have recently faced difficulties in the labor market 
and are most likely to benefit from employment and training 
assistance. As a result, with a pre-post design It is difficult to 
determine whether the observed post-program earnings gains should be 
attributed to training or are merely an artifact of the way in which 
individuals are selected Into the program. 

Because a randomly assigned control group is not feasible and a before- 
and-after comparison Is not appropriate, a comparison group must be 
drawn from a population of program-eligible Individuals who do not 
participate in JTPA. The major disadvantage of using nonpartlcipants 
as a comparison group is the potential for selection bias. 
Specifically, individuals who choose to participate in JTPA may be 
systematically different from those who are also eligible, but do not 
choose to participate. To the extent that these differences can be 
measured, they can be controlled for in the analysis. However, there 
may also be unmeasured differences, such as motivation or attitude 
toward work, that affect both whether individuals participate In JTPA 
as well as their subsequent employment experiences. A comparison of 
the outcomes of participants with the outcomes of nonpartlcipants risks 
attributing the effects of these pre-program differences to JTPA. This 
Indicates the Importance of selecting a comparison group that is as 
similar as possible on measured and unmeasured characteristics to the 
sample of JTPA participants. In the remainder of this section we 
elaborate on the criteria that should be used in choosing a comparison 
group of nonpartlcipants and discuss the advantages and disadvantages 
of various alternative comparison groups. 




Criteria for Choosing a Comparison Group of 
Program-Eligible Nonparticipants 

The two major considerations Involved 1n chooslr.g a comparison group 
are the size of the group and Its similarity to the participant 
sample. A large comparison group 1s desirable because 1t enables more 
efficient or precise estimates of net Impacts. Consequently 
comparison groups that can be obtained less exoenslvely are more 
desirable than comparison groups that requ're expensive data collection 
because a larger sample may be available within a fixH budget. 

The comparison group should also be similar to the participant ^roup on 
both observed and unobserved characteristics. Similarity on observed 
characteristics Increases the efficiency of the estimation of net 
Impacts because the correlation between Individual characteristics „nd 
JTPA participation 1s small. In addition, the estimates of net impacts 
are likely to be less affected Ly unmodeled nonl 1near1t1es and 
Interactions. That 1s, when the distribution of observed variables 1s 
similar 1n the two groups, specification errors affect both the 
comparison and the participant samples 1n the same manner, so that the 
estimated difference 1n their behavior should not be affected. 

It 1s especially Important for the comparison group to be similar to 
the participant sample 1n terms of characteristics that are related to 
eligibility for JTPA. According to the JTPA legislation, to be 
eligible for Title II-A programs, adults must be 22 years of age or 
older and be economically disadvantaged. The legislation defines 
economically disadvantaged to mean an Individual who: (1) 1s a timber 
of a family that in the six months prior to application received a 
total Income of less than the OHB poverty level or less than 70% of Uie 
lower living standard Income "evel, whichever is greater, given th^ 
person's family size; (2) 1s a member of a family that receives 
federal, state, or local cash welfare payments; (3) 1s receiving food 
stamps; (4) is a foster child for whom state or local support payments 
are made; or (5) is a handUapped individual who is economically 
disadvantaged but whose family 1s not, a<> permitted by the Secretary of 
Labor. The Act requires that at least 90% of Title II-A participants 
be economically disadvantaged and allows up to 10% of the participants 
to be individuals who are not economically disadvantaged provided they 
have encountered barriers to employment (e.g., limited English language 
proficiency, school dropouts, ex-offenders). 27 In addition, the Act 
requires that recipients of AFDC grants and school dropouts be 
equitably served in relation to their incidence 1n the eligible 
population. To the extent possible, the comparison group should only 
include individuals who meet the explicit eligibility criteria and who 
are similar to participants on characteristics emphasized in the 
legislation. 



It appears, however, that relatively little use is being made of 
the 10% "window" for serving persons who are not economically 
disadvantaged. For example, Cook et al. (1985) report that 94% of all 
persons served by Title II-A programs during the transition year were 
economically disadvantaged. 



It 1s also desirable for the participant and comparison groups to be 
similar in unmeasured characteristics so that selection bias can be 
minimized. However, such similarity 1s extremely difficult to 
achieve. One approach 1s to measure some aspects of normally 
unobserved characteristics, for example, through questionnaire Items 
that measure motivation or work ethic. Because the net Impact study 
will rely exclusively on agency data, however, this approach 1s not 
feasible. Other approaches Involve either choosing comparison group 
members who have gone through selection processes . s1m1 lar to those used 
to screen JTPA applicants, or utilizing complex statistical models 1n 
an attempt to correct for potential differences 1n unmeasured 
characteristics. Alternative approaches to deal with selection bias 
are discussed 1n Chapter 5. 



Alternative Comparison Groups of Nonparticipants 

In this section we discuss alternative comparison groups of 
nonparticipants that can be drawn from existing data bases. As 
described above, the primary advantage of drawing a comparison group 
from an existing data base 1s that 1t 1s relatively Inexpensive and 
allowr. a greater sample size for a given budget, thus resulting 1n 
greater statistical power. The major disadvantage of using an existing 
data base 1s that the analysis must necessarily be limited to variables 
contained 1n both the JTPA MIS and 1n the data base for the comparison 
group. To be useful 1n the JTPA net Impact model, the Information 1n 
the data base for the comparison group should pertain to the same 
general time period as the data collected for participants, and the 
common baseline variables should be measured comparably. In addition, 
1t 1s Important that comparison group members have not received 
employment and training program assistance 1n the past or that 
variables measuring the extent of treatment be available. 

We considered several alternative data bases for use 1n developing a 
comparison group Including: 

• Various national surveys (e.g., the Current Population Survey, 
the Survey of Income and Program Participation, the DOL National 
Longitudinal Survey); 

• State files of UI claimants; 

• JTPA applicants who are "no-shows*; and 

• State files of Employment Service (tS) registrants. 

Below wp discuss the advantages and disadvantages of each of these 
alternative comparison group strategies. 

Various National Surveys. Although most previous studies of the net 
Impact of employment ar.d training programs used comparison groups drawn 
from various national data bases, such an approach 1s simply not 
feasible ror « state-level net Impact model. This 1s primarily because 
thise data sources do not have local area Identifiers, which are 
necessary to ensure that the comparison group 1s drawn from the same 
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state and local area as the participant sample. Moreover, even 1f this 
problem could be overcome, these data sets are deficient In that they 
do not have sufficient sample sizes at the state and local level. 
Finally, the data sets are generally not available 1n a timely fashion. 

UI Claimants. An advantage of developing a comparison group from the 
file of UI claimants Is that 1t 1s possible to ensure that the 
comparison group members are from the same state and local areas as 
participants. In addition, there are a few demographic characteristics 
such as age, race, sex, and occupation on the data file that could be 
used as control variables In the model. A sample of UI claimants, 
however, has two Important drawbacks. First, claimants are very 
different from JTPA participants In terms of their work histories 
That only approximately 10X cf adult JTPA participants are UI claimants 
supports this view. Second, because claimants are generally eligible 
to receive UI benefits for some time longer than the average length of 
treatment In JTPA, such a comparison group would artificially Inflate 
the net Impact estimate on earnings. 28 

JTPA Program No-Shows. JTPA applicants who are eligible and assigned 
to a program activity but who do not part1dpate--"no-shows"--have 
certain potential advantages as a source for drawing a comparison 
group. The primary advantage of using these Individuals as a 
comparison group Is that they underwent many of the selection processes 
that participants did. Thus, they were eligible, decided to apply for 
JTPA, were selected to participate, and were also assigned to a program 
activity. In addition, a major potential advantage Is that because 
these Individuals applied to JTPA, comparable baseline data were 
collected for them, which maximizes the amount of pre-program 
Information that could be used to construct control variables for the 
net Impact model. The major disadvantages are that (1) these groups 
Introduce an additional selection bias In that they may be 
systematically different because they chose not to participate (perhaps 
because they found a job); (2) the samples sizes of such groups may be 
too small to support precise net Impact estimates; and (3) data on 
no-shows are not usually Included In the JTPA MIS. Thus, although this 
approach has certain advantages, It raises additional questions and 
cannot be Implemented unless states and SDAs consistently Include 
no-shows In the data base. 

ES Registrants. Drawing a comparison group of ES registrants In 
offices In the areas served by the SDAs has several advantages. First, 
data are available on several Individual characteristics of Interest, 
which should generally be comparably measured In the JTPA MIS and, as 
such, can be Included as control variables. Except for some potential 
exceptions noted below, this usually Includes the variables used to 
define eligibility for JTPA. Second, like JTPA participants, ES 
registrants also presumably experienced a recent decline In earnings. 



He also rejected the Continuous Wage and Benefit History 
(CWBH)— a spedal-usa subfile of UI claimants available In certain 
states that has detailed work history 1nformat1on--because It Is 
available only for claimants and only In a very limited number of 
states. 
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Finally, also like JTPA participants, ES registrants are 1n the labor 
force at the time they applied for assistance. As discussed 1n 
Chapter 2 f It 1s Important to ensure that participants and comparison 
group members are similar In 'their attachment to the labor force, or 
else net Impact estimates can be severely biased. 

Although ES registrants should be more similar to JTPA participants 
than Individuals 1n other data sources, the use of ES registrants as a 
source for drawing a comparison group has certain disadvantages. For 
example, to the extent that ES registrants receive other employment and 
training services, the comparison group would look less like an 
"untreated" group. If ES registrants receive significant employment 
assistance, a comparison of the average post-program outcomes of JTPA 
participants and ES registrants would measure the Incremental effect of 
JTPA relative to the ES, and not the effect of JTPA per se. Although 
this Is not an uninteresting question to address, particularly at a 
time of limited resources for employment and training program 
assistance, there are other procedures that can be used to deal with 
this problem and estimate the net Impact of JTPA. For example, one 
could obtain data on ES services received and control for the effect of 
those services In the net Impact model. 29 Alternatively, one could 
restrict the sample of ES registrants to persons who did not receive ES 
services. Because a substantial proportion of ES registrants generally 
receive no services other than registration, there are likely to be a 
sufficient number of ES registrants that receive no services and that 
are similar to JTPA participants on measured characteristics that could 
be used to develop a comparison group. ^ 

A second potential problem In using ES registrants to develop a 
comparison group concerns the possibility that the ES registrant file 
may be dominated by UI claimants In some areas. For example, In states 
and local areas In which ES offices are co-located with UI offices, 
this could result In a large proportion of UI claimants In the ES 
registrant file. As we argued above, It would be Inappropriate to 
compare the outcomes of JTPA participants with a sample that Is 
dominated by UI claimants. Thus, In Instances where ES offices are 
co-located with UI offices, It may be necessary to undersample UI 
claimants In the ES registrant file to obtain a comparison sample that 
has a proportion of claimants similar to the JTPA population. 



It must also be recognized that the ES Is a potential source of 
job placement assistance for some JTPA termlnees. As such, It could be 
Important to control for ES services received by participants as well 
as by comparison group members. 

^n 

s - u Because the net Impact model relies on a regression approach with 
dummy variables representing the treatments, the estimated Impact 
ultimately Involves a comparison of the earnings of participants and of 
ES registrants who receive no ES services. As such, the estimated 
Impact should be similar whethsr one controls for ES services 1n the 
model or Instead excludes Individuals who receive ES services, it 1s 
Important to note, however, that the power of the analysis 1s 
considerably greater for a given sample size If those who received ES 
services are first excluded from the analysis. 



A third potential disadvantage to using ES registrants concerns recent 
reductions 1n federal reporting requirements related to the ES that 
will result 1n less consistent Information across states. In 
particular, states are TO. longer required to report the number of 
economically disadvantaged applicants who are registered and served by 
the ES. Because economically disadvantaged status 1s the major 
criterion for JTPA eligibility, and given the Importance of ensuring 
that the comparison group be similar to participants on all 
characteristics that affect eligibility, 1t 1s Important that the 
economically disadvantaged status variable be available for the net 
Impact model. It appears, however, that many states have decided to 
continue collecting Information on the economically disadvantaged 
status of ES registrants 1n the event that reporting requirements 
change again. For states that no longer collect this Information, 1n 
order to Implement the net Impact model they will either have to change 
their data collection practices or Implement more complex procedures to 
draw a comparison group that 1s matched to participants on the level of 
earnings and recent changes 1n pre-program earnings, similar to the 
approach used by Westat (1982) and others to evaluate the net Impact of 
CETA programs. 

A final potential disadvantage to using ES registrants as a source for 
drawing comparison groups concerns differences across states 1n their 
procedures for retaining historical data. In the past, most states 
have kept copies of data tapes with Individual ES records (Including 
registrant characteristics and ES services received) for a period of 
three to five years. In some states, however, Individual -level data 
are purged after one year. In such states 1t would not be possible to 
conduct the analysis retrospectively, since data on comparison group 
members for Individuals who enrolled 1n JTPA early 1n the year may 
already have been purged. Thus, such states would have to draw the 
comparison samples on an ongoing basis or alter their practices 
concerning the retention df historical data. 

Despite these potential disadvantages, we believe that ES registrants 
are the best source for drawing a comparison group to examine the net 
Impact of JTPA. As a result, 1n the remainder of this section we 
discuss the variables that are comparably measured 1n the ES and JTPA 
MIS data and that can be Included as control variables In the model. 
In subsequent sections, we discuss additional details related to 
drawing a sample of ES registrants. Including the time frame for 
selection, cases that should be excluded, and procedures for drawing 
the sample. 

In Appendix A, we provide copies of the application forms that ar* 
currently being used 1n the State of Washington for ES registration and 
for JTPA application. Based on a review of these forms, the following 
data Hems appear to be comparably measured and «< nich can reliably be 
Included In the net Impact model: 

• Age (1n years) ; 

• Race (White, Black, Hispanic, American Inrtlan/Alaskar. Native, 
As1an/Pac1f1c Islander); 



• Sex; 



• Education (whether received high school degree or equivalent); 

• Handicapped status (whether has physical or mental Impairment 
that Is a substantial handicap to employment); 

• Occupation (primary DOT code of previous job); 

• Veteran status (whether a veteran, whether a Vietnam-era 
veteran, and whether a disabled veteran); 

« Food Stamps recipient; 

• VJIN registrant; end 

• Economically disadvantaged status. 



In addition to these Items that should be available and comparably 
measured for participants and comparison group members In the JTPA awl 
ES MIS systems, other state agency d*ta bases will provide key control 
variables for the net impact model. Specifically, from various UJ data 
bases and from the PA data base one will obtain comparable measures of: 

• ^re-program quarterly earnings; 

• Pre-program UI benefit payments received; and 

• Pre-prorram AFDC grants received, 

£1 though these two lists provide considerable Information on Individual 
a: cter^tlcs In the preprogram period that can be used 1n the net 
ta#act model, the Informat on Is not as complete as one would Ideally 
fU::j. In particular, 1t ,ould be desirable to have measures of marital 
sj;*tus, family size, dependent children, ex-offender status, limited 
fellsh-speak^ng ab11H w * f and detailed data on pre-program employment 
experiences. It mure be recognized, however, that most of these 
ch* acterlstlcs were also unavailable to previous national studies of 
r)--4 Impact of gove;?irnent subsidized employment and training programs, 
*s such, this 1s r.:A a limitation that 1s specific to the state-level 
Model . 



Comparison Gt zi> Exc ^> 

Prior to selecting the comparison group of ES registrants, certain 
cases shou r 1 be excised from the sample frame to maintain 
comparabllV y Kfith the participant sample. In addition, 1t is 
necessary i:\ xclude from the comparison group Individuals who a*e 
clearly not eligible for JTPA. it may also be desirable to exclude 
those who are likely to have earnings functions that are considerably 
different from the earnings functions of JTPA participants. Below we 
discuss sample exclusion considerations concerning the comparison group 
of ES registrants. 

M§_§i ? J Hissing Data, To maintain comparability with the JTPA 
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participant sample, the group of ES registrants should be restricted to 
individuals who are at least age 22 and who are under 65 years of age. 
The ES registrant sample should also be restricted to cases that do not 
have missing data on key Independent variables. 

JTPA Participation. It 1s Important to exclude ES registrants from 
the sample who are JTPA participants during either the pre-program, 
Inprogram or post-program period. This problem, known as "comparison 
group contamination," results 1n comparing the outcomes of JTPA 
participants with the outcomes of comparison group members who also 
participated 1n JTPA and yields net Impact estimates that are biased 
toward zero. To minimize this problem, one should compare the social 
security numbers of current and recent JTPA participants with the 
social security numbers of ES registrants and exclude all matches from 
the comparison sample. 

kgfegr Force Attachment. As discussed 1n Chapter 2, most recent 
smhas have generally Imposed a labor force attachment criterion on 
comparison sample member.. Because ES registrants are 1n the labor 
force at the time they apply to the Job Service, however, 1t 1s 
possible to align the comparison group so that the sample of ES 
registrants 1s selected at approximately the same time as JTPA 
participants are enrolling 1n JTPA. Specifically, the comparison group 
should be drawn from new ES registrants 1n the same calendar quarter 
that participants enroll 1n JTPA. Following this procedure, other 
registrants who applied for ES assistance more than three months 
earlier and, as such, experienced their decline 1n earnings even 
earlier and who may have already returned to work, will be excluded 
from the sample frame for the comparison group. 

II ■ Se L r .y 1ces - Another Issue concerns potential exclusions to ensure 
that the comparison group 1s as untreated a: possible. In particular, 
in addition to excluding from the sample frame potential comparison 
group members who have participated 1n or are participating 1n JTPA, 1t 
Is also Important to exclude Individuals who receive significant ES 
services. The major ES services Include ^ferritl to jobs listed with 
t h !l* f?' , Job devel °P ment » Job counseling, arsd testing. Because 
individuals who receive such services may be receiving considerable 
employment assistance, we recommend that such Individuals be excluded 
from the comparison group sample frame. Since Individuals may receive 
some ES services several months after they apply for assistance, we 
recommend that a period six months after application be used for 
determining whether a person received significant ES assistance. 
Because a large proportion of ES registrants generally receive no 
services other than registration during a six-month, post application 
period (Johnson, Dickinson, and West, 1985), there are likely to be a 
sufficient number of ES registrants who receive no services and who are 
similar to JTPA participants on measured characteristics to develop a 
comparison group. 

JTPA Eligibility. A final Important sample exclusion Issue concerns 
procedures to ensure the similarity of the participants and comparison 
group members on characteristics related to eligibility for JTPA. As 
Indicated above, the primary criterion for JTPA eligibility 1s that the 
person be economically disadvantaged. Although the Act requires that 
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at least 90% of JTPA participants be economically disadvantaged, In PY 
1984, 95% of adults 1n Title II-A programs were economically 
disadvantaged. Moreover, of those who are not economically 
disadvantaged, or who cannot be certified to be economically 
disadvantaged, the Act requires that they face demonstrated employment 
barriers. Because virtually all adult Title II-A enrollees are 
economically disadvantaged, and even those who are not certified to be 
economically disadvantaged may In fact be so, or may have earnings 
potential that Is most similar to economically disadvantaged 
Individuals, we recommend that the comparison group also exclude all 
new ES registrants who are not economically disadvantaged at 
application. This will ensure a comparison group that Is similar to 
JTPA participants on the key characteristic related to JTPA 
eligibility. 31 



Selecting the Comparison Group sample 

Once decisions have been made on which Individuals should be excluded 
from the comparison sample frame, the next step Involves procedures to 
use for drawing the comparison group. We considered the following 
three alternative approaches for selecting the comparison group: 

• Including all ES registrants In the sample frame; 

• Developing a matched comparison group; and 

• Using a subsample of ES registrants that has the same 
distribution as JTPA participants on a few key characteristics 
related to JTPA eligibility. 



As we describe below, each of these approaches has advantages and 
disadvantages In terms of the resources required to conduct the net 
Impact analysis and the validity of the results. These tradeoffs must 
be kept In mind In Identifying the most appropriate procedure for a 
state-level net Impact evaluation. 

Including All ES Registrants . The first alternative has the advantage 
of maximizing the size of the comparison group and thus the statistical 
power of the analysis. It has a major disadvantage, however, of 
generating a comparison group with demographic characteristics and 
employment histories that are very different from those of the 



01 As described In Chapter 2, several previous studies excluded from 
the comparison group individuals with very high pre-program earnings 
who were clearly Ineligible to participate in employment and training 
programs . By matching participants and comparison group members on 
economically disadvantaged status, however, such additional exclusions 
are no longer necessary. It should b«e noted that 1f a comparable 
measure of economically disadvantaged status 1s not available In some 
states for ES registrants, then procedures to exclude cases with high 
pre-program earnings (e.g., one standard deviation above the maximum 
earnings of participants) would have to be Implemented. 
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Srilrlnill J ?\.« fj , a result ' the correlation between JTPA 
participation and Individual characteristics 1s likely to bo relatively 
high, and the anticipated gain 1n statistical power may not be 
realized. In addition, as discussed earlier, using the entire 
compar son sample could result 1n biased net Impact results because of 
unspecified nonl1near1t1es and Interactions that arise from estimating 
the regression model over a wide range of Incomes. As such, we do not 
recommend this approach. 

Hatched Comparison Group. An alternative approach Involves choosing a 
comparison group from the sample frame that 1s specifically matched to 
the participant sample on the basis of characteristics that are related 
to program participation and the post-program outcomes of Interest 
This procedure has the Important advantage of reducing the pre-program 
differences between the participant and comparison samples so that 
regression estimates are less sensitive to unspecified nonl 1near1ties 
and Interactions. However, the sample size would be greatly reduced, 
which would result 1n lower statistical power. Moreover, the principal 
matching approaches that have been used 1n developing comparison groups 

* *f« 1 Jf at1ng i A he 1mpact of em Ployment and training programs- 
stratified matching and nearest-neighbor matching— are complex 
procedures to Implement and require considerable staff and computer 
resources. In addition, unless a one-to-one match Is developed, the 
Impact analysis would have to be weighted, which would be an additional 
complication. Because of the excessive resources Involved 1n 
conducting the analysis with a comparison group that Is specifically 
matched to the participant sample, we believe 1t 1s unrealistic to 
develop a matched comparison group for the state-level net Impact model. 

Matching Distributions on Kev variable 0 ur recommended approach 1s 
a compromise between the two alternatives discussed above. 
Specifically, we recommend that comparison group members be selected 
from the sample frame to ensure that the distribution of the resulting 
comparison group 1s similar to the distribution of participants on key 
characteristics. This approach maintains maximum statistical power to 
the extent possible, while ensuring that the participant and comparison 
samples are similar on key characteristics. As Indicated above, 
because of program rl1g1bH1ty considerations and certain practical 
Issues concerning the relationship between the ES and other programs 
such as UI and welfare, some of the more Important characteristics on 
which to ensure comparability between participants and comparison group' 
members are economically disadvantaged status, receipt of UI, and 
receipt of AFOC. Because we will ensure comparability between the two 
groups on economically disadvantaged status by excluding from the 
sample frame for the comparison group all new ES registrants who are 
not disadvantaged, no additional matching 1s required on this 
characteristic. Although we do not have any Information on the 
proportion of new ES registrants that are AFOC recipients or that are 
UI recipients nationwide, there are likely to be relatively too few ES 
registrants who are AF0C rec1p1ents (e g f 9 percent Qf aduU men an(J 

35 percent of adult women JTPA termlnees In PY 1984 were receiving AFOC 
at application) and relatively too many ES registrants who are 
receiving UI benefits (e.g., 15 percent of adult men and 8 percent of 
adult women JTPA termlnees 1n PY 1984 were UI claimants at 
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application). In order to ensure similarity on these Important 
characteristic we recommend that comparison group members be randomly 
selected from cue sample frames of adult men and women to match the 
distribution of participants on these characteristics. Thus, for the 
separate samples of adult men and women, procedures would be used to 
ensure that the participant and comparison groups are similar on the 
proportions In the four cells representing combinations of AFDC and UI 
recipiency status. Operationally, for a given total sample size of 
participants and comparison group members, sampling rates for each cell 
would be determined to match the two distributions and then comparison 
group members would be selected randomly from the cells at the given 
sampling rates. Alternatively, the maximum combined sample size could 
be obtained by Including all comparison group members In the cell with 
the smaU?st ratio of ES to JTPA cases and then adjusting the sampling 
rate for the other three cells to obtain the same proportion of 
comparison group members as participants In each cell. 



SAMPLE SIZES FOR THE PARTICIPANT AND COMPARISON SAMPLES 

An Important element of the research design Is the determination of the 
appropriate sample sizes for the participant and comparison groups. As 
we Indicated earlier, In many states there will be little choice 
Involving the size of the participant sample. That Is, It may be 
necessary to use all JTPA participants to obtain reasonably precise 
estimates of average program net impacts. In such cases, the only 
issue Involves the size cf the program impacts that can be detected at 
a given statistical power for different sized comparison group 
samples. Provided sufficient staff and computer resources, one should 
generally use the largest sizeo comparison group feasible. Because the 
marginal cost of increasing sample size Is very low since data are 
available In existing agency files, even in medium to large states one 
should generally use the largest numbers of participants and comparison 
group members feasible, given available staff and computer resources. 
In states with large JTPA programs, however, samples of participants 
and comparison group members will be drawn, which raises the Issue of 
total sample size as well as the allocation of the total sample among 
the two groups. In the remainder of this section we describe an 
approach to guide states in selecting appropriate sample sizes for the 
net impact model . 

In order to make an Informed decision concerning both the total sample 
size and the allocation of the total sample between the participant and 
comparison* groups. Information on several factors is required. For 
example, the appropriate sample size for the net impact model 



" For example, based on State of Washington data for the first nine 
months of PY 1985 (July 1, 1984 - March 31 , 1985), of the 360,000 ES 
registrants active during this period, only 7% were welfare recipient* 
and nearly 50% were UI claimants. On the other hand, nearly 30% of 
Title II-A terminees 1n Washington during this period were welfare 
recipients and less than 15% were UI claimants. For the most part, ES 
registrants and JTPA participants were reasonably similar on other 
demographic characteristics such as age, race, and education. 
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ultimately depends on the size of the Impact that 1s Important to 
?!n«SL J!!l ? Purposes and the level of statistical accuracy 
required, with larger sample sizes, one has greater assurance of 
detecting small differences 1n overall outcomes between the participant 
and comparison groups, as well as detecting differences for major 
participant subgroups or across program activities. The likelihood of 
detecting a given difference In outcomes also depends on the allocation 
of the total sample between the participant and comparison groups and 
the unexplained variance of the outcome measure (I.e., earnings, AFDC 
grants). in general, the more homogeneous the samples (1 e the 
smaller the variance of the outcome measure), the smaller will be the 
number of cases necessary to detect a given difference 1n outcomes at a 
specific level of significance. Below we Illustrate how some of these 
factors can affect the sample size required for testing various 
hypotheses concerning the net Impact of JTPA on earnings. 33 

In Tables 3 through 6 we provide Information on the likely precision or 
estimates of annual earnings Impacts for different sized samples of 
JTPA participants and comparison group members, separately for adult 
men and women. These estimates of the precision of annual earnings 
Impacts are based on simple d1fference-1n-means tests and assume that 
the unexplained variance of earnings 1s the same for participants and 
comparison group members. 34 The final two columns of these tables 
indicate the ilkely observed Increase 1n earnings due to JTPA that 
would be statistically significantly different from zero at least 90X 
of the time. That 1s, one would have 90% power at a .10 significance 
level of detecting an overall Increase 1n JTPA participants' annual 

Although the sample size required for examining the net Impact of 
JTPA on welfare dependency 1s likely to be different, Information on 
the unexplained variance of AFDC grants was not available to estimate 
sample sizes with any degree of confidence. We expect, however, that 
sample sizes that are adequate for examining earnings Impacts will also 
be adequate for examining Impacts on AFDC grants. 

34 Although the analysis models described 1n Chapter 5 rely on 
murtlple regression techniques 1n which the hypothesis that JTPA 
Increases earnings 1s examined by performing a "t"-test on the JTPA 
participation dummy variable, because the comparison group of ES 
registrants will be similar to JTPA participants on key characteristics 
that affect eligibility, the t-test 1s essentially a difference 1n 
means test. As such, stands procedures can be used to calculate the 
standard deviation of a difference 1n means test. The standard 
deviation of a simple difference in means test 1s given by 

S = ( V N P + S c /N c )1/2 ' " here S p and s c are the variances of earnings for 
the participant and comparison groups respectively, and N D and N c 

Js n I! te K t i e « sam i ) J e s1 f es for the two aroups, which are shown In Tables 3 
through 6 1n the columns headed "Relevant Sample Sizes." Based on an 
analysis of earnings of CETA participants, we estimate the unexplained 
variance of earnings for participants and comparison group members 
would be approximately (*6, 000)2 for adult men an(J ($4 700) 2 f or 
adult women. 
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Table 3 



PRECISION OF ESTIMATES OF ANNUAL EARNINGS 
IMPACTS FOR AVERAGE STATE WITH EQUAL 
SIZED COMPARISON GROUPS 

Observed Impact Required 
Standard Deviation of for 90J Power at .10 

rnMPARKHMQ f Re ! eva ! t Mn Dffference Significance Level 
WAW Sample Sizes Ten MuT —jfe Tggr~ 

I. JM vs. Compartson Group 2 



. Overall Impact 


3,000 


3,000 


$155 


$121 


$397 


$310 


. CT Impact 


1,200 


3,000 


205 


161 


526 


413 


. OJT Impact 


750 


3,000 


245 


192 


628 


492 


u . JSA Impact 


600 


3,000 


268 


210 


687 


538 


II. Within-program impacts' 5 














. CT compared to other activities 


1,200 


1,800 


224 


175 


574 


513 


. High school graduates compared 
to nongraduates 


2,250 


750 


253 


198 


741 


580 


. Blacks compared to other 
race/ethnic groups 


900 


2,100 


239 


187 


700 


547 


. Female welfare recipients 
compared to nonrecipients 


1,200 


1,800 




175 


mmm 


513 



Notes: a - A one-tail significance test is assumed, 
b - A two-tail significance test is assumed. 
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Table 4 



PRECISION OF ESTIMATES OF ANNUAL EARNINGS 
IMPACTS FOR AVERAGE STATE WITH 
EXPANDED COMPARISON GROUPS 



Relevant 



Standard Deviation of 
Mean Difference 



Observed Impact Required 
for 90S Power at .10 



OMPARISONS 


Sample Sizes 


Men 


Women 


Hen 


Women 


TPA vs. Comparison Group 3 














Overal 1 Impact 


3,000 


9,000 


$126 


$ 99 


$323 


$254 


CT Impact 


1,200 


9,000 


184 


144 


472 


369 


OJT Impact 


750 


9,000 


228 


179 


585 


459 


JSA Impact 


600 


9,000 


253 


198 


649 


508 


ithin-progrsm impacts 0 














CT compared to other activities 


1,200 


1,800 


224 


175 


574 


513 


High school graduates compared 
to nongraduates 


2,250 


750 


253 


198 


741 


580 


Blacks compared to other 
race/ethnic groups 


900 


2,100 


239 


187 


700 


547 


Female welfare recipients 
compared to nonrecipients 


1,200 


1,800 




175 




513 



a - A one-tail significance test is assumed, 
b - A two-tail significance test is assumed. 
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Table 5 



PRECISION OF ESTIMATES OF ANNUAL EARNINGS 
IMPACTS FOR LARGE STATE 



Relevant 



Standard Deviation of 
Mean Difference 



Observed Impact Required 
for 90* Power at JO 



COMPARISONS 


Sample Sizes 


Men 


Women 


Men 


Women 


JTPA vs. Comparison Group 3 














. Overall Impact 


9,000 


9,000 


$ 89 


$ 70 


$228 


$179 


. CT Impact 


3,600 


9,000 


no 


93 


302 


238 


, OJT Impact 


2,250 


9,000 


141 


111 


362 


285 


. JSA Impact 


1,800 


9,000 


155 


121 


397 


310 


Within-program impacts h 














. CT compared to other activities 


3,600 


5,400 


129 


101 


378 


296 


. High school graduates compared 
to nongraduates 


6,750 


2,250 


146 


114 


427 


334 


. Blacks compared to other 
race/ethnic groups 


2,700 


6,300 


138 


108 


404 


316 


. Female welfare recipients 
compared to nonrecipients 


3,600 


5,400 




101 




296 



es: a - A one-tail significance test is assumed, 
b - A two-tail significance test is assumed. 
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Table 6 



PRECISION OF ESTIMATES OF ANNUAL EARNINGS 
IMPACTS FOR SMALL STATE 



COMPARISONS 

JTPA vs. Comparison Group a 
. Overall Impact 

• CT Iapact 

• OJT Impact 

• JSA Impact 

rfi thin-program impacts^ 

, CT compared to other activities 

i High school graduates compared 
to nongraduates 

i Blacks compared to other 
race/ethnic groups 

, Female welfare recipients 
compared to nonrecipients 



Relevant 



Standard Deviation of 
Mean Difference 



Observed Impact Required 
for 90* Tower at .10 
Significance Usel 



Sample Sizes 


Men 


Women 


Hen 


Women 


1,000 


1,000 


$267 


$210 


$ 685 


$ 538 


400 


1,000 


354 


279 


90B 


715 


250 


1,000 


423 


333 


1,085 


& 


200 


1,000 


465 


363 


1 s 192 




400 


600 


387 


303 


1,134 


80<: 


750 


250 


438 




1,281 


1,002 


300 


700 


414 


324 


1,212 


948 


400 


600 




303 




888 



a - A one-tail significance test 1s assumed, 
b - A two-tail significance test 1s assumed. 
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earnings as s^aU as the values given 1n these two columns. 35 Before 
discussing ths Information provided 1n these tables, 1t must be 
emphasized that these estimates depend on the unexplained variance of 
earnings for participants and comparison group members and, If the 
samples ultimately selected are more homogeneous , one would be able to 
etect smaller Increases In earnings for a given total sample size at 
the same power. 

In Table 3 we present estimates of the precision of the *!<nual earnings 
gains that would be obtained from a state with an average size JTPA 
program and with equal numbers of participants and comparison group 
members. In particular, the Illustration \n Table 3 assumes 6,000 
adult JTPA termlnees equally divided between men and women. As this 
table Indicates, with 3,000 adult male JTPA participants and 3,000 
adult male ES registrants serving as the comparison group, the standard 
devletlon of the mean difference 1n earnings between the two groups 1s 
estimated to be $155 (see formula described 1n footnote 34) . This 
Indicates that an average gain of approximately $400 In annual earnings 
due to JTPA would be statistically significant at least 90% of the 
time. For adult women, we estimate that somewhat trailer average JTPA 
Impacts (approximately $300) can be detected with sample sizes of 3,000 
participants and 3,000 comparison group members. This Is because of 
the smaller expected unexplained variance In earnings for adult women. 
Bassd on estimates of the average earnings for adult men and women, 
these results Indicate that a five (six) percentage point effect on 
earnings can be detected for men (women) v;Hh 90% power at a .10 
significance level. 36 

The remainder of Table 3 Illustrates how the precision of net Impact 
estimates differs for various subgroup , 37 As expected, the minimum 
size* of wlthln-prograifi Impacts that can be detected with the same 
degree o? statistical accuracy <*re considerably larger. For example, 
It 1s only possible to detect an average Increase 1n earnings due to 
JSA participation jf approximately $700 ($540) for men (women) at least 
r 054 of the time, kith this sample design U Is also possible to detect 
s1» : ?ar Impacts for blacks and h^gh school graduates. 



Th* statistical power Is the probability of detecting an effect 
at the chosen significance If^el when an effect of the specified size 
In fact exists Formally, U Is equal to 1 minus the probability of 
r.akiing a Type II error. The use of 90% power requires that the 
hypothesis test be correct 90% of the time. 

3f The estimates of average annual earnings for adult men of $8,600 
and for jdult women of "'5,200 are derived from data for 1976 CETA 
enrc^ees and rested to reflect Inflation since that time. 

37 The relevant sample size estimates reported 1n these tables 
assume a randof sample of JTPA termlnees. Specifically, based on data 
for JTPA trrmlnees 'hi the nine-month transition period, we assume that 
40* of aduV* termlnees are assigned to classroom training, 25% to 0JT 
and 20% to job search assistance. In addition, high school graduates 
are assumed to comprise 75% of termlnees, blacks 30%, and 40% of the 
female termlnees are assumed to be welfare recipients. 
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In Table 4 we demonstrate the sensitivity of the precision of n*t 
Impact estimates to expanding the size of the comparison group. In 
particular, we maintain a participant sample size of 3,000 each for 
adult men and women and Increase the size of the comparison group to 
9,000 each. As Table 4 Indicates, e xpanding the comparison group 
considerably doe<= not appreciably reduce the size of earnings Impacts 
that can be dettrted with 9 OX power at a .10 significance level , for 
example, the overall annual impact that can be detected for men 
declines from $397 to $323, and for women from $310 to $254. The value 
of the additional comparison group members 1s even less for estimating 
separate subgroup Impacts. 

In Tables 5 and 6 we describe tne precision of estimated earnings 
Impacts for large and small state JTPA programs respectively. As Table 
5 Indicates, by Increasing the participant sample to 9,000 and 
maintaining a comparison group sample of 9,000, 1t 1s possible to 
detect small overall effects and moderate sized effects for several 
subgroups, for example, with such a sample design, we estimate that 
one could detect an anr"*l earnings impact of as small as $228 for men 
and $179 for women, which correspond to approximately 2.5 to 3 
percentage point Impacts. Moreover, such a sample provides the same 
precision for estimating the separate Impact of Job search assistance 
programs as a sample size of 3,000 comparison group members and 3,000 
participants provides 1n examining the overall Impact of JTPA, 

The results 1n Table 6 reveal the difficulty small states will have 1n 
obtaining precise estimates of the Impact of JTPA programs. Because 
several states are likely to have fewer than 1,000 adult male enrollees 
and 1,000 adult female enrollees 1n Title II-A programs 1n a given 
program year, should such states undertake a net Impact analysis the 
results must be carefully Interpreted. That 1s, as Table 6 Indicates, 
because of the small sample sizes available, "ily very large net 
Impacts can be reliably detected--on the order of $1,000 or 
more— Impacts that are considerably larger than those generally 
reported 1n previous studies. As such, these states are very likely to 
find net Impacts that are not statistically significantly different 
from zero at conventional levels of significance. In such 
circumstances, states should refrain from concluding that JTPA has no 
Impact and focus on the signs and magnitudes of the estimated program 
Impacts. 

The above discussion has Illustrated how the minimum program Impact 
that can be detected with 90% power at a .10 significance level varies 
by sample size and the sample allocation between participants and 
comparison group members. However, the question remains as to the 
appropriate sample size and allocation for conducting a state-level 
JTPA net Impact analysis. Provided one retains the criteria of 90% 
power and a .10 significance *evel, for a given unexplained variance 1n 
the outcome measure the cho ce of sample size ultimately rests on the 
minimum detectable effect that 1s considered to be acceptable, for the 
most part, however, this 1s a matter of Judgment, and little guidance 
can be provided. 

One possible approach, recently offered by Stafford (1985), Involves 
selecting a sample size such that the minimum detectable post-program 
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effect covers the average cost of JTPA programs. Stafford suggested a 
post-program time horizon of four years as reasonable 1n order to 
capture the longer-term earnings gains of participants compared to 
non participants. 3 * We Implemented this general approach using a 
discount rate of 5% and assuming JTPA costs are similar to the 
experiences during the nine-month transition period, In which the 
average cost per termlnee was approximately $2,300. Given equal sizes 
of the participant and comparison groups for men and women, In order to 
yield a combined average discounted effect of $2,300, this suggests 
that the Impact for men that can be estimated at 90% power Is 
approximately $700 per year, and the Impact for women Is approximately 
$550. As the results 1n Table 6 Indicate, a sample size of 1,000 
comparison group members and 1,000 participants each for adult men and 
women would meet this criterion for the overall Impact of JTPA, but 
none of the subgroup Impacts would be estimated with any degree of 
precision. On the other hand, based on the results In Table 3, with 
3,000 persons In each cell one could ensure that the minimum detectable 
effect covered average program costs for all major subgroups. It 
should also be noted that samples of approximately 6,000 adult men and 
5,000 adult women, divided equally between participants and 
nonpartldpants, were used In several recent studies to examine the net 
Impact of CETA on earnings (see, for example, Westat, 1982; Dickinson, 
Johnson, and West, 1986). 

Thus, we believe that a total sample of 12,000— divided equally between 
adult men and women, participants and comparison group members (I.e., 
3,000 each)— should be adequate to meet most states' analysis 
needs. 39 States that are Interested In obtaining precise net Impact 
estimates for subgroups of adult men or women, or that have additional 
resources should consider larger sample sizes as necessary. Finally, 
states with relatively small JTPA programs (I.e., less than 2,000 adult 
enrollees per year) should be very careful In Interpreting the results 
as only very large Impacts will be regarded as significantly different 
from zero. As a result, such states might consider pooling samples 
with other states or Increasing the sample size over time to enhance 
the reliability of the net Impact findings. 



M It should be recalled that Ashenfelter (1978) reports program 
Impacts for women that persist over a five-year post-program period and 
for men that decline somewhat over the five-year period. 

39 It should also be noted that with such a sample design 1t will be 
possible to detect approximately a 2.4 (2.8) percentage point Impact on 
the probability of employment 1n the post-program year for adult men 
(adult women) with 90% power at a .10 significance level. These 
estimates assume that 83% of the adult men and 75% of the adult women 
have earnings 1n the post-program year, which are based on data for 
CETA participants. 
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CHAPTER 5. DATA ANALYSIS PLAN 



The final step 1n developing a model to obtain valid estimates of the 
net Impacts of JTPA programs on the earnings and welfare dependency of 
adult men and women Involves the specification of a data analysis 
plan. In particular, a data analysis strategy must be developed for 
examining the adequacy of the comparison groups selected and for using 
the comparison groups to estimate the net Impacts of JTPA on 
participants' post-program outcomes. In this chapter, we describe an 
overall estimation strategy for obtaining valid estimates of the net 
Impacts of JTPA. 

As described 1n Chapter 4, our proposed approach Involves a comparison 
of the post-program labor market experiences of a sample of JTPA 
participants with the experiences of a sample of economically 
disadvantaged ES registrants who do not receive ES services. If the 
samples of JTPA participants and ES registrants are similar on both 
measured (e.g., age, race, education) and unmeasured (e.g., attitude 
toward work, motivation) characteristics, then valid Inferences about 
the Impacts of JTPA programs can be drawn from such comparisons. 
However, whether an Individual decides to participate 1n JTPA or 
decides to register with the ES Is likely to depend on both Individual 
and agency decisions, and because of these selection processes the two 
groups may differ on a number of dimensions that could result 1n biased 
net Impact estimates. This Is the Issue of selection bias. 

It 1s Important to recognize that virtually all nonexperlmental 
approaches will contain a certain amount of bias. That 1s, the formal 
conditions required to ensure unbiased estimates of program effects are 
not likely to be met, even 1f one had extensive data on the 
characteristics of program participants and comparison group members. 
This 1s particularly true for the proposed research design, which does 
not Involve detailed survey data and, as a result, the amount of 
Information available to control for selection bias 1s limited. As 
such, the fundamental Issue 1s not whether any bias exists, but how 
large 1t 1s, whether some bounds can be placed on the likely direction 
and magnitude of the bias, and whether the assumptions under which 1t 
1s possible to correct for the bias are at all realistic. 

To provide Information on such Issues requires a well-designed analysis 
strategy and a careful Interpretation of the results that recognizes 
the Inherent limitations of the basic approach. Below we describe an 
analysis strategy for examining the adequacy of the comparison group 
selected to get a better understanding of the likely direction and 
magnitude of selection bias. We then descMbe alternative statistical 
models for estimating the average net Impacts of JTPA and Impacts for 
Important subgroups, and Indicate the conditions under which the 
various models will produce unbiased estimates of program Impacts. The 
chapter concludes with a discussion of measurement error and other 
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statistical Issues, and of potential adjustments for certain data and 
design deficiencies. 



Before describing the analysis strategy, 1t 1s Important to note that 
some of the material presented 1n this chapter 1s quite technical. 
Although we have attempted to make the discussion as accessible as 
possible to practitioners, a certain amount of technical presentation 
and statistical jargon 1s unavoidable. As such, program administrators 
will need to discuss Issues of concern with Individuals on the1> staffs 
who are qualified to carry out a net Impact analysis. Those who have 
difficulty with the technical material should also consider consulting 
the Implementation manual for the net Impact model (Volume VI) which 
discusses the recommended approach In a somewhat less formal manner. 



EXAMINING THE ADEQUACY OF THE COMPARISON GROUPS 

OBTAINING EVIDENCE ON SELECTION BIAS 

As Indicated above, the most important threat to obtaining valid 
estimates of the net Impacts of JTPA programs Is the potential of 
selection bias, that Is, systematic differences between the participant 
and comparison groups. Although a group of local ES registrants who do 
not receive ES services Is likely to be the best comparison group 
available (given the limited resources states have for evaluation 
activities), It must be recognized that several selection processes 
affect who Is Included In the JTPA and ES groups. For example, JTPA 
participants must decide to apply to the program; they must meet 
certain legislated eligibility criteria; they must be selected by the 
agency for program participation and assigned to a program; and they 
must decide to accept that assignment and enroll In the program. 
Although ES registrants do not have to meet any formal eligibility 
criteria, certain individuals, such as those receiving benefits from 
UI, are required to register with the ES, and some offices follow 
selective registration policies. Furthermore, whether an ES registrant 
receives ES services depends on several factors, Including the 
availability of suitable job openings and the qualifications and 
persistence of the applicant. Because of these various selection 
processes, it 1s unlikely that the resulting samples of JTPA 
participants and ES registrants who do not receive services are 
equivalent on all measured and unmeasured characteristics that affect 
labor market outcomes. As a practical matter, ther fore, one should 
not focus on the fact that the two groups are not identical , but 
identify the major dimensions on which the groups differ; determine the 
extent to which the net impact estimates are likely to be sensitive to 
those differences; and adjust statistically for those differences to 
the extent that data and analysis resources allow. 

In this section we describe three different criteria that can be usad 
to judge the adequacy of the comparison groups selected and outline 
some recommended analyses to determine whether these criteria are 
met. 40 It should be noted that the criteria and analyses recommended 



It should be noted that the analyses described below could also 
be used as input to choosing among alternative potential comparison 



here are very similar to those used by Westat (1982), Dickinson 
Johnson, and West (1985, 1986), and Fraker and Haynard (1984) to 
evaluate the quality of matched comparison groups for estimating the 
net Impacts of various employment and training programs. Before 
describing these analyses, however, 1t 1s Important to recognize their 
Inherent limitations. The fundamental limitation 1s that these 
analyses provide evidence only on the comparability of the two groups 
In the pre-program period, whereas the crucial assumptions for deriving 
unbiased net Impact estimates concern the comparability of the two 
groups 1n the post-program period, which, of course, are Inherently 
untenable. That 1s, these criteria are necessary, but not sufficient, 
conditions for the comparison groups to overcome the problem of 
selection bias. Thus, e\.»n 1f the two groups are comparable 1n the 
pre-program period, this should not be Interpreted as definitive 
evidence that there 1s no selection bias, since changes 1n the 
underlying behavioral relationships could have occurred from the pre- 
to post-program period. Nevertheless, the analyses described below 
should provide useful Information on the extent and types of 
differences between participants and comparison group members that must 
be kept 1n mind when Interpreting the net Impact results. 



Criterion One: Similarity in Measured Characteristics 

The first criterion that we recommend be used to judge the 

comparability of the participant and comparison groups 1s the 

similarity of the two groups on measured characteristics during the 

pre-program period. For example, one could compare the means and the 

distributions of measured characteristics for participants and 

comparison group members, separately for adult men and women.* 1 It 



groups. That Is, because the choice of an appropriate comparison group 
Is not entirely clear a priori, such analyses could be used to 
determine, for example, whether all ES registrants or JTPA no-shows 
would potentially be better comparison groups than ES registrants who 
receive no services. 

4 \ w ° uld als0 bfe possible to estimate an OLS linear probability 
model of the likelihood of participating 1n JTPA to determine the major 
differences between the two groups. That 1s, one would estimate a 
regression equation with the dependent variable equal to 1 for JTPA 
participants and 0 for comparison group members, and the Independent 
variables would be all measured characteristics Included 1n the net 
Impact model described later In this chapter. This approach has the 
advantage of estimating the Independent Influence of each measured 
characteristic, while controlling for the Influence of all other 
characteristics, which eliminates the confounding effects of other 
variables that mj be present when comparing mean characteristics. 
That 1s, a comparison of mean characteristics could Indicate, for 
example, that JTPA participants are more likely to be minorities and 
less educated, whereas the regression approach would account for the 
differences 1n education by race and could reveal that, after adjusting 
for differences 1n race, there are no differences between participants 
and comparison group members in terms of education levels. 
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1s particularly Important to compare the participant and comparison 
groups on available measured characteristics that are known to affect 
earnings and AFDC grants. For example, because of the well-known 
differences 1n earnings and AFOC participation by race, one would 
compare the participant and comparison groups In terms of the 
proportion white, black, and Hispanic. Although differences In 
measured characteristics such as race can be controlled for directly 1n 
the net Impact model and, as such, should aot necessarily be of primary 
concern, major differences 1n measured characteristics between the two 
groups could be Indicative of a selection process that also holds for 
unmeasured characteristics, which would cause a more serious analytical 
problem. For example, 1f one found that the JTPA samples of adult men 
and women had much larger proportions of minorities and high school 
dropouts than the samples of ES registrants, It could be that JTPA 
participants were also more disadvantaged on unmeasured characteristics 
(I.e., less motivated). This would result In underestimating the net 
Impacts of JTPA. 

It would also be useful to conduct similar analyses across JTPA program 
activities. That Is, one would compare the characteristics of 
participants separately by progress activity (e.g., CT, OJT, JSA). This 
would provide some Indication of the additional selection bias that 
could arise In estimating separate net Impacts by program activity. 
For example, 1f 1t were determined that more advantaged Individuals 
were being sent to OJT, the net Impacts of this program activity would 
be somewhat Inflated because of this assignment process. That 1s, the 
observed positive relationship between earnings and the OJT treatment 
would not be entirely causal, but In part would reflect the fact that 
OJT participants were more advantaged (I.e., more motivated). On the 
other hand, If there were relatively few differences In measured 
characteristics by program activity, this evidence, 1n combination with 
other tests described below, would provide some, confidence that no 
additional selection biases would be Introduced In deriving estimates 
of the net Impacts by program activity. 



Criterion Two: Similarity in Preprogram Earnings and 
AFDC Grants 

The second criterion we recommend be used to judge tt»e adequacy of the 
comparison groups 1s the similarity of the pre-png.v^ earnings and 
AFDC grants of participants and comparison gro* ; Members. This 
Involves a test of whether there 1s a significant difference 1n the 
pre-program earnings and AFDC grants of the two groups, controlling for 
measured characteristics, Such a test provides valuable evidence on 
whether the two groups are comparable on the basis of the lagged 
dependent variables (after controlling for measured characteristics) 
or, alternatively, whether there are differences 1n the outcome 
variables between the groups 1n the pre-program period that are due to 
unmeasured characteristics. If there are any differences In adjusted 
pre-program earnings or AFDC grants between the two groups, th^n this 
analysis will also provide some hints a? to the direction and magnitude 
of the selection bias. For example, the extent to which JTPA 
participants have lar*ur (smaller) adjusted pre-program earnings than 
ES registrants provides some Indication as to whether they are more 
(less) advantaged on the basis of unmeasured characteristics, and the 
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size of the difference 1s a reasonable estimate of the amount by which 
the net program Impacts could be overstated (understated) 1f the 
difference persisted in the post-program period. 

To formally test for differences 1n the pre-progrem earnings (or AfDC 
grants) of participants and comparison group members, one would 
estimate a regression equation (separately for adult men and women) 
with pre-program earnings (or AFDC grants) as the dependent variable. 
For example, one could estimate the following equation using ordinary 
least squares (OLS) regression techniques: 

(1) Y i,t-r a ♦ b \t-i + vz i ♦ A Vi,t-j ♦ m \t-i + cp i,t ♦ e i,t-r 

where t 1s the period of enrollment Into JTPA; Y 1 t ^ represents the 
earnings of Individual 1 in the Immediate pre-program period t-1 ■ 
x 1,t-l represent? tha vector of exogenous characteristics (e.g., age, 
race, education) Included 1n the net Impact model and measured just 
prior to enrollment 1n tra'i1ng;42 1s a vector of Interaction 
terms (e.g., between race aid other Independent variables 1n the 
model); Y 1f t _j represents the earnings of Individual 1 1n earlier 
pre-program periods t-j; H 1 t _j represents labor market 
characteristics facing 1nd1v1dua! "1 1n the pre-program period; t 
1s 1 for JTPA participants and 0 f »-»r comparison group members; e^ [ -i 
1s a random error term; and a, b, v„ dj, m and c are coefficients' to 
be estimated. The test for differences in pre-program earnings between 
the participant and comparison groups 1s based on a t-test of the 
estimated coefficient c, where the t-rat1o 1s given by the estimated 
coefficient divided by Us standard error. 43 Given that the sample 
sizes Involved are likely to be large, the hypothesis of no difference 
between the two groups 1n pre-program earnings or AFDC grants would be 
rejected at the .05 (.01) significance level 1f the calculated t-rat1o 
exceeded 1.96 (2.58) 1n absolute value. 

On a more Intuitive level, because participation 1n JTPA during period 
t can not have an effect on earnings In period t-1, the estimate of c 



It should be noted that some of the t-1 characteristics 
Included 1n the net Impact model below should not' be Included 1n this 
pre-program earnings equation because the variables are jointly 
determined with pre-program earnings. For example, the Food Stamps 
dummy variable and the female WIN registrant dummy should be excluded 
from the pre-program earnings equation. 

One could also estimate a similar pre-program earnings or AFDC 
grants equation and replace the JTPA participation dummy variable with 
a set of variables representing different program activities to provide 
Information on the differences 1n pre-program earnings across program 
activities that are due to unmeasured characteristics. By testing 
whether the separate program activity coefficients are significantly 
different from zero, this would Indicate whether there are likely to be 
any additional selection biases due to the nonrandom assignment of 
programs to Individuals that will affect the net Impact estimates by 
program activity that are described later 1n this chapter. 
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1n the model described above should be 0. The extent to which c 
deviates frosn 0 provides evidence on the direction and magnitude of the 
I1k*ly selection bias. That 1s, large negative (positive) values of c 
Indicate that participants were less (more) advantaged than comparison 
group members 1n the pre-program period on unmeasured characteristics, 
and If this persisted through the post-program period, It would likely 
result In understating (overstating) the net Impact of JTPA. Thus, for 
example, If this analysis Indicated that, after adjusting for differ- 
ences 1n measured characteristics, the pre-program earnings of JTPA 
participants were $200 less (more) than the earnings of the comparison 
group, then one might consider adding (subtracting) $200 to (from) the 
net Impact estimate to adjust for differences In unmeasured character- 
istics. It should be noted, however, that because pre-program earnings 
and AFDC grants will be Included as Independent variables In some net 
Impact models, the extent of this bias shoufd be less In the post- 
program period. As such, adjdstlng the net Impact estimate for the 
total difference In pre-program earnings Is likely to overcompensate 
for the bias due to differences In unmeasured characteristics. 

It should be noted that with several periods of pre-program earnings 
data, 1t Is possible to examine In more detail the . o^irablllty of the 
earnings dynamics In the pre-program period. 4* In particular, one 
could estimate a regression equation like the one described above for 
each pre-program period and derive a vector of estimated coefficients 
of c. The only difference In the equations would be that earnings 1n 
subsequent pre-program periods would have to be omitted from each equa- 
tion (I.e., the equation for Y^_ 3 would not Include t _ 2 or 
Y 1,t-1 as explanatory variables). f One would expect that the extent 
of any bias In program Impacts would tend to zero as a longer history 
of pre-program earnings Is corrected for In the regression. That Is, 
the largest (1n absolute value) estimated coefficients of c should 
occur 1n the very early pre-program periods, and the coefficients 
should tend toward zero as additional lagged values of earnings are 
Included 1n the equation. 45 



Criterion Three: Similarity of Preprogram Earnings 
Equations 

The third criterion that we recommend be used to judge the adequacy of 
the comparison groups Is the similarity of the earnings equations for 



* q An alternative method of evaluating the similarity of the earn- 
ings dynamics of the participant and comparison groups Involves 
comparing the correlation of earnings over time for the two groups In 
the pre-program period. See Cooley, HcGuIre, and Prescott (1979) for a 
discussion of this approach. 

45 Such a pattern was observed in ar. evaluation of the Impact of ES 
job referrals on earnings by Johnson, Dickinson, and West (1985) for 
men, but was not true for women, for whom the estimated values of c 
varied little across the different pre-program periods. Thus, they 
concluded that Including a long series of lagged earnings accounted for 
much of the potential selection bias for men, but not for women. 
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JTPA participants and for the comparison groups In the pre-program 
period. 4& This criterion, which 1s considerably more strict than the 
previous two, Is quite Important because, 1f the same model 1s 
generating earnings 1n the two groups, 1t suggests that program Impacts 
will not be sensitive to nonl Inearl ties 1n the earnings equation. This 
would provide additional confidence In our ability to obtain unbiased 
estimates of program Impacts. 

To test for differences 1n the pre-program earnings equations of 
participants and comparison group members, one would estimate separate 
regression models for the two groups, as well as a regression model for 
the two groups combined. These models would be estimated separately 
for adult men and women. For example, to test whether the pre-program 
earnings models were the same for the two groups In period t-1 , one 
would estimate the following equations using OLS regression techniques: 

Participant Eodel : 



k 

P ,t-i - "p ' Vp,t-i T Vp,t-i T jb u pr Pf t-j T Vp.t-i T * Pf t-r 



(2a) Y . , = a + b X ,.,+vZ , , ♦ d t Y . , + m M , , + e 



Comparison Group Model : 

k 

(2b) Y = a + b X . + v Z + E d .Y . . + m M . . + e . lf 
c,t-l c c c,t-l c c,t-l J=2 cj c,t-j c c,t-l c,t-l f 

and 



Combined Model : 

(2C) Y lfW = a ♦ bX lfW * vZ^ ♦ i y,^ ♦ * ♦ . 



where subscript p (c) refers to participants (ccmparlson group 
members), and the third equation 1s estimated over all Individuals (1), 
participants and comparison group members combined. Similar equations 
could be estimated for earlier pre-program periods to determine whether 
the earnings models are the same 1n these periods as well. 

The test that the earnings equations are the same for the two groups 1s 
equivalent to testing that the effects of the measured characteristics 
on earnings are similar for the two groups. To perform this test, one 
must compute the following test statistic: 



* 6 As we describe 1n a later section, the net Impact models assume 
that this criterion 1s met 1n the post-program period. 
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[RSS - (RSS c +RSS p )]/r 



(RSS c +RSS p )/(N-K) 



where RSS Is the residual sum of squares from the combined regress 5<x 
equation, RSS p (RSS C ) Is the residual sum of squares from v 
regression over participants (comparison group members) only, r Is th : 
number of restrictions Involved (I.e., the number of Indepencten* 
variables 1n each equation), and N-K Is the number of degrees of 
freedom w!ien no restrictions are Imposed (I.e., the sum of the number 
of degrees of freedom from the separate participant and comparison 
group models). Under the assumption that the error terms are normally 
distributed, the test statistic given above follows Snedecor's (1956) 
F-d1str1but1on with r degrees of freedom In the numerator and N-K 
degrees of freedom In the denominator. If the test statistic exceeded 
the critical value for the specified level of confidence, then the null 
hypothesis would be rejected (I.e., we would conclude that the earnings 
models for the two groups are not similar). 47 For example, suppose 
the earnings model had 10 Independent variables and a total sample size 
of 6,000. Then, there would be 10 degrees of freedom In the numerator 
and 5,980 degrees of freedom In the denominator, so the test statistic 
would follow a F(10, 5,980) distribution. Given the critical values 
for the F-dlstrlbutlcn with these degrees of freedom, one would 
conclude that the earnings models of the two grcups were different at 
the .05 (.01) significance level If the test statistic exceeded 1.83 
(2.32). 



How to Interpret Evidence on the Adequacy of the 
Comparison Groups 

The three criteria and related analyses described above should provide 
considerable Information regarding the adequacy of the comparison 
groups 1n the pre-program period and the types of likely biases that 
must be dealt with. It should be emphasized again, that these criteria 
are relatively strict tests of the comparability of the two groups and 
one should not generally expect nonexper1mentally-der1 ved comparison 
groups to meet all of them. If the conditions are generally satisfied, 
then the chances of obtaining unbiased program net Impact estimates 
using the standard statistical models described below are considerably 
Improved. If the criteria are strongly rejected (e.g., F-stat1st1cs of 
10 or 20 when approximately 1.5 1s sufficient for rejection), then one 
should be very careful 1n proceeding to estimate net Impacts with these 
comparison groups. Instead, one should first double check to ensure 
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47 An alternative method of conducting this test 1s available and 
can be performed with most standard software packages. Specifically, 
one would estimate an equation that Included all of the explanatory 
variables 1n Equation (2c), plus each of the variables multiplied by 
the JTPA participation dummy variable. The formal test of whether 
earnings 1n the pre-program period are different for the two groups 1s 
based on an F-test of the hypothesis that the coefficients of the 
Interaction terms are jointly Insignificant. This 1s sometimes 
referred to as a M Chow w test. 
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that the data processing and analysis guidelines described earlier W re 
followed and, 1f the criteria are still strongly rejected, one should 
then consider obtaining assistance from a researcher familiar with 
these Issues. If, as 1s most likely, the results are somewhere In 
between (I.e., pre-prograro differences between the two groups that are 
sometimes statistically significant, but not exceptionally large), then 
one win need to understand the Implications of these differences for 
Interpreting and adjusting the net Impact's results. 



ESTIMATING THE NET IMPACTS OF JTPA PROGRAMS 

We now turn to a description of alternative statistical models for 
estimating the average net Impacts of JTPA programs on participants' 
post-program outcomes. In describing the alternative models, we 
attempt to carefully indicate the conditions under which earn model 
will result 1n unbiased estimates of program effects. Based on the 
advantages and disadvantages of the models, we present a recommended 
model. Using the recommended statistical model, a subsequent section 
describes how to obtain separate estimates of net Impacts for major 
demographic groups and by program activity. It should be noted that 
although the specific models described below may not be familiar to 
state-level analysts, each model can be estimated using standard 
estimation techniques contained 1n statistical software packages (e.g., 
SAS, SPSS* that should be readily available and familiar to them. 

The statistical models presented below describe how to derive estimates 
of the net Impacts of JTPA on post-program earnings. It should be 
emphasized, however, that this Is for Illustrative purposes only. To 
obtain estimates of the net Impacts of JTPA on the receipt of AFDC 
grants by Individuals, the only change required would be to replace the 
earnings variable In a particular period with AFDC grants 1n that 
period; the other variables to be tested for Inclusion 1n preliminary 
models would be the same. The same general models would also be 
estimated to determine the net Impacts of JTPA on the probability of 
employment or on the probability of AFDC participation. Depending on 
the specific outcome measure, however, some additional statistical 
Issues arise. These Issues are discussed 1n a later section. 

It should be noted that some of the net Impact models described below 
Impose restrictions on the nature of the earnings generating process. 
Because these restrictions are more likely to be valid for "real" 
earnings (I.e., earnings expressed 1n constant dollar terms) than for 
nominal or money earnings, the earnings variables should be converted 
Into constant dollar values by deflating them with the Consumer Price 
Index. This would be particularly Important 1f the recent pattern of 
very low Inflation rates (I.e., 4% per year) were to end and the 
double-digit Inflation rates of the late 1970s returned. Because there 
are no valid price Indices available at the state level, as Indicated 
above we recommend that the BLS Consumer Price Index for All Urban 
Consumers be used to deflate nominal earnings and create measures of 
real earnings. 48 
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For example, for a given nominal earnings series, Yt, 
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Before describing the alternative net Impact models, 1t 1s Important to 
briefly discuss the choice of post-program periods to be examined. In 
estimating the net Impacts of JTPA on post-program outcomes, one can 
either define the outcomes 1n terms of specific calendar periods, or 1n 
terms of specific periods after program termination. Virtually all 
previous studies of the net Impacts of employment and training programs 
used earnings 1n a given time period, e.g., a specific calendar year, 
as the outcome measure for both participants and comparison group 
members. These studies also generally used enrollee-based participant 
samples (I.e., samples of participants who enrolled 1n the program 
during a given time period), and created a "pre-program period" that 
was common for all Individuals and that was defined to end at the 
beginning of the enrollment period. As a result of these 
methodological choices, the estimated net Impacts capture earnings 
changes from a pre-program period to a post-program period that 
corresponds to the Identical calendar months/years for all participants 
and comparison group members. Because of differences 1n program length 
of stay, however, 1t must be recognized that such net Impact estimates 
do not reflect Impacts for a common period since termination. 
Consistent with this tradition, the models described below can produce 
net Impact estimates for several different points during the 
post-program period, but these periods will not generally correspond to 
a similar period after termination for all participants unless there 1s 
little variation 1n program length of stay. 

There 1s likely, however, to be considerable program and policy 
interest 1n estimates of net Impacts that correspond to fixed periods 
after termination (e.g., first three months after termination, first 
year after termination). This 1s 1n part because of the recent 
development of JTPA follow up systems 1n many states, that track the 
employment status of participants at fixed points after termination, 
usually at three months and sometimes again later. Although the UI 
earnings data do not allow one to create outcome measures that 
precisely correspond to a given period after termination, 1t 1s 
possible to create earnings measures for various three-month periods 
after the calendar quarter of program termination. Thus, for 
Individuals who terminate 1n a given calendar quarter, earnings 
received during the next calendar quarter, that are from three to six 
months after termination (4.5 months on average), could be used to 
approximate the Impacts of JTPA for a common period after termination. 

An Important conceptual problem arises, however, 1n estimating net 
Impacts for a fixed period after termination. This 1s because the 
concept of "date of termination" has no meaning for comparison group 
members and, as a result, additional complexities arise. In 



Y t-1 »••-» Y t-k» anQ * a corresponding consumer price Index series 
CPIt. CPI t _i CPI t _k, with CPI t =100, then a real earnings 

series 1n constant period t dollar values 1s given by Y t /CPIt 
Jt-l/CPIt-l. Y t _ k /CPI t _ k . If, Instead, CPI t _ 1= 100, then 

It Is necessary to first create a revised Index with CPIt=100— by 
dividing tte entire CPI series by CPI t — and then using the revised 
serlss as deflators 1n order to obtain a real earnings series 1n 
constant period t dollar values. 
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particular, to minimize biases using enrol lee -based samples, one would 
have to select the "post-program periods" for the comparison group to 
match the distribution of participants' termination dates across 
calendar quarters. For example, suppose that of the Individuals who 
enrolled 1n JTPA 1 ri the first quarter of 1986, 35 percent terminated 1n 
the first quarter, 50 percent terminated 1n the second quarter, and 15 
percent terminated 1n the third quarter of 1986. Then, 1n estimating 
the net impact of JTPA on earnings during the 3 months after the 
calendar quarter of program termination, the real earnings outcome 
measure for participants would be based on earnings during the second 
quarter of 1986 for 35 percent of the sample, earnings during the third 
quarter for 50 percent of the sample, and earnings during the fourth 
quarter for 15 percent of the sample. To avoid biases, the comparison 
group would have to be aligned similarly. That is, the 35, 50, and 15% 
distributions across quartets would have to occur for the comparison 
group as well. Although this post-program alignment problem could be 
avoided if a termlnee-based sample of participants were used (I.e., 
since the period .jfter termination would be common for all participants 
and comparison group members), this would create a comparable alignment 
problem in the pre-program period. That is, beu of differences in 
length of participation, participants would have enrolled in JTPA over 
several previous quarters, and comparison group members would have to 
match this distribution 1n the pre-program period. 

Thus, although 1t is possible to estimate net Impacts for outcome 
measures that approximate the Impacts for common post-termination 
periods, we do not believe the additional data processing and analysis 
complications are worth the effort. Moreover, some of the econometric 
models we describe require that the outcomes be measured for the same 
calendar periods for all participants and comparison group members. As 
a result of these considerations, the net Impact models described below 
are based on outcome measures and Independent variables that are 
defined for the same calendar periods (both pre-program and 
post-program) for both participants and comparison group members. 



Autoregressive Earnings Model 

Several recent nonexperimental evaluations of the net Impacts of 
employment and training programs (e.g., westat, 1982; Dickinson, 
Johnson, and West, 1986; and Gerad, 1984) have primarily relied on 
analys1s-of-covariance approaches that are variants of Ashenfelter's 
(1978) autoregressive earnings model. Using this approach, these 
studies included as Independent variables 1n the regression equation 
the pre-program earnings history and all socioeconomic characteristics 
that were comparably measured for participants and comparison group 
members. In addition, Interaction terms were also Included in order to 
control as much as possible for potential differences between the two 
groups. Thus, an autoregressive earnings model that would be 
applicable for evaluating the net impacts of JTPA at the state level 
would have the following general form: 
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where t 1s the period of training; Yj fS 1s the earnings of Individual 
1 1n a post-program period s; t-1 ,s a vector of pre-program 
socioeconomic characteristics of Individual 1 f where the t-1 subscript 
Is Included to emphasize that the characteristics are measured at 
application to JTPA (and at a similar time for ES registrants); Is 
a vector of Interaction terms (e.g., between age and race, education 
and race, pre-program earning* and race) to account for the fact that 
the effect of various characteristics on earnings may differ by race or 
personal characteristics; t _j are the earnings of Individual 1 1n 
various pre-program periods {-j, J»l....,k; Hj s Is a vector of labor 
market characteristics applicable to Individual 1 In post-program 
period s (all Individuals located In the area served by a given SDA 
would have the same value during this period) ; t Is a dummy 
variable that Is 1 for JTPA participants and 0 for comparison group 
members; e^ s Is an error term; and a, b, v, d* , m and c are 
parameters t'o be estimated. Such a model could be estimated using 
ordinary least squares regression techniques, and separately for 
different post-program periods (s) to obtain Information on the timing 
of the net Impacts for a given cohort of trainees or for several 
cohorts combined. 

With such an autoregressl ve model estimated separately for adult men 
and women, the estimated coefficient of the JTPA participant dummy 
variable (c) represents the average net Impact of JTPA on the outcome 
measure for the particular post-program period (s) for both groups. 
For dependent variables that are expressed In dollar terms such as 
earnings and AFDC grants, the coefficient of the JTPA participant dummy 
variable can be Interpreted as the average dollar Impact on the 
particular outcome measure. By dividing the estimated dollar Impact by 
the mean earnings or AFDC grants of comparison group members, this 
prov lies an estimate of the percentage change In earnings or AFDC 
grants due to JTPA. 

For dummy dependent variables such as whether employed 1n a particular 
period, or whether receiving AFDC grants during a particular period, 
the autoregressl ve net Impact model Is equivalent to a linear 
probability model. That Is, the model essentially estimates the 
effects of various factors on the probability of a certain event 
occurring (e.g., having positive earnings 1n a given post-program 
period). As such, the estimated coefficient of the JTPA participant 
dummy variable can be Interpreted as the average percentage point 
change 1n the probability of working or receiving AFDC grants due to 
JTPA. Once again, by dividing the estimated percentage point change by 
the mean proportion of comparison group members, one can obtain an 
estimate of the percentage change 1n the probability of working (or 
receiving AFDC) due to JTPA. 

With this model , the estimated coefficient of c will provide an 
unbiased estimate of the average net Impact of JTPA on ^ost-program 
earnings 1n period s provided two conditions are met: (1) all 
characteristics that affect earnings and program participation are 
Included 1n the model, so that the correlation between the error term 
and the participation dummy variable Is zero, and (2) the earnings 
model that would prevail 1n the absence of training 1s the same for 
JTPA participants and comparison group members (I.e., the third 
criterion for the adequacy of the comparison group described earlier 
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Sit ??t * J" f Je Post-program period). « The first condition- 
that all factors that affect program participation and the outcome 
measure be Included 1n the model— 1s equivalent to saying that any 
unmeasured characteristics that affect earnings are uncorrelated with 

P ^,A m A P art J, c1 P at J°"- 50 , Thus . ™ the measured characteristics 
included In the net Impact model capture the factors that affect the 
program participation selection process, 1t 1s possible to obtain 
unbiased net Impact estimates of program effects using an 
autoregresslve earnings model. For example, Can (1975) has shown that 
If selection Into the program 1s determined solely on the basis of 
pre-program earnings (or other measured characteristics 1n the model) 
then consistent net Impact estimates can be obtained from an 
autoregresslve model. This 1s true even 1f the error term 1n the 
earnings equation (e 1>s ) 1s autoregresslve (as 1s likely to be the 
case), as long as the error term 1n the program participation equation 
Is uncorrelated with e 1fS . Thus, although 1t 1s well known that the 
coefficients of the lagged earnings variables will be biased 1f the 
error term 1s autoregresslve, 1t 1s possible to derive consistent 
estimates of net program effects from such an autoregresslve model. 

An advantage of an autoregresslve earnings model like the one described 
above that places no restrictions on the relation between current and 
prior earnings (except for linearity) 1s that 1t controls for 
differences 1n measured characteristics between JTPA participants and 
comparison group members that remain after the matched comparison group 
Is selected. At the same time, however, because of the limitations of 
the proposed research design discussed earlier, the Independent 
variables that are comparably measured for JTPA participants and ES 
registrants and that are likely to be available 1n most states for 
Inclusion as control variables (I.e., X, t _ } ) 1n the net Impact model 
are severely limited. For example, based'on data currently Included 1n 
the various program HIS systems for the State of Washington, the 
following demographic/socioeconomic variables could be created and 
Included 1n the net Impact model: 

• Age (1n years) ; 

• Age-squared 5 ^ ; 



In addition, we are assuming that all variables 1n the model are 
perfectly measured. The consequences of measurement error are 
discussed later 1n this chapter. 

50 Note that this condition does not require that all factors that 
affect earnings must be Included 1n the model, but only those that 
affect both program participation ard earnings. If some character- 
istics that only affect earnings (but not program participation) are 
not measured, and therefore omitted from the model, then the 
coefficients of other control variables could be biased, but not c. 

51 Age-squared 1s Included to capture the nonlinear relationship 
between earnings and age that has been documented extensively 1n the 
literature. An alternative procedure would be to create mutually 
exclusive and exhaustive age categories (e.g., 22-24, 25-29, 30-34, 
35-39, 40-44, 45 and older) and replacing age and age-squared with 
dummy variables for the age categories selected. 
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• Black dummy variable (1 If Black; 0 otherwise); 

• Hispanic dummy variable (1 1f Hispanic; 0 otherwise); 

• Other nonwhlte race dummy variable (1 1f not Black, Hispanic, or 
White; 0 otherwise); 

• Grade 12 dummy variable (1 1f completed high school only; 0 
otherwise) ; 

• Grade 13+ dummy variable (1 If attended some college; 0 
otherwise) ; 

• Handicapped dummy variable (1 If person has a physical or mental 
Impairment that Is a substantial handicap to employment; 0 
otherwise) ; 

• Hale veteran dummy variable (1 If male and served 1n the Armed 
Forces; 0 otherwise) ; 

• Food Stamps dummy variable (1 If receiving Food Stamps at 
application; 0 otherwise); 

• Female WIN registrant dummy variable (1 If female and a WIN 
registrant at time of application; 0 otherwise); 

• UI payments received In the pre-program calendar quarter; 

• 1 -digit occupation dummy variables (1 1f occupation prior to 
application Is a specific 1 -digit DOT code; 0 otherwise). 
Treating professional and managerial occupations as the left-out 
category, the occupational dummies to be Included would be: 



-clerical and sales 
-service 

-agriculture, fishing, and forestry 

-processing 

-machine 

-benchwork 

-structural, and 

-miscellaneous. 



Although this is a fairly extensive 11st of demographic and 
socioeconomic characteristics, It is far from ideal. Specifically, 
these variables would not allow one to directly control for differences 
1n marital status, family size, presence of dependent children, 
ex-offender status, limited English speaking ability, or pre-program 
employment intensity such as total hours worked or total weeks worked. 
To the extent that these unmeasured factors affect program 
participation, as well as earnings, then the autoregressi ve earnings 
model described above could produce biased estimates of net program 
impacts. At the same time, however, it Is Important to keep in mind 
that many of these characteristics were also not available to previous 
national studies of the Impact of employment and training programs, so 
that this potential limitation Is not unique to a state-level model . 
Moreover, the proposed model has an advantage over previous studies 
that partially overcomes this data deficiency 1n that, in addition to 
the demographic/socioeconomic characteristics listed above, the model 
will also include quarterly values of pre-program earnings (or AFDC 
grants) covering as long a period as data permit (e.g., up to 12 
quarters 1f possible). These quarterly histories will enable one to 
better control for pre-prcgram differences In earnings or welfare 
dependency patterns that could In part reflect differences 1n 
unmeasured characteristics between the participant and comparison 
groups and, as such, will help to reduce the problem of selection tias. 
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!S«Vhi!, t1C V nal advanta9 * of the state-level model 1s that It will be 

Prelltlt n a °t1 O n 0 a n i ^h 0 ' d1fferenc « 1n ^or ™rket conditions 
Previous national studies were unable to control for labor market 
variables 1n examining the net impacts of employment and training 
ErZ amS 1 1 the 9 9 °9 ra P h1c location of participants and comparison 
group members was unknown, which prevented the linking of labor market 
data to specific Individuals. The number of different measures of 
labor market conditions that can be Included 1n the autoregresslve 
earnings model will primarily depend on the data available, the number 
of SDAs 1n the state (since all persons who reside 1n the service area 
of a given SDA will receive the same value for the local labor market 
characteristic In a particular period), and the extent to which the 
local labor markets differ across SDAs. At a minimum, 1t 1s desirable 
to Include the local unemployment rate as a control variable 1n the 
autoregresslve earnings model. 
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Fixed-Effects Models 

As described above, the autoregresslve earnings model controls for 
differences 1n measured characteristics between participants and 
comparison group members to the extent possible and can yield 
consistent estimates of program net Inpacts under certain conditions 
However, such models cannot control for differences in unmeasured 
characteristics (e.g., ability, motivation) between the two groups that 
may be introduced by the various selection processes involved. That 
is, if Individuals either self-select Into JTPA or are selected by 
program operators on the basis of unmeasured characteristics, and these 
characteristics affect earnings, this would Introduce a correlation 
between the error term and the participation dummy variable 1n the net 
Impact model, which wouJd lead to biased estimates of program effects. 

; „ « ct1on we describe an alternative model— known as a 
fixed-effects model-that under certain conditions can overcome 
problems caused by differences in unmeasured characteristics between 
participants and comparison group members. 

The f Jxed-effects model, used by Bassl (1983, 1984), Kiefer (1979), and 
Ashenfelter (1978), explicitly recognizes that an Individual's earnings 
and the decision to participate 1n JTPA are likely to depend on 
unmeasured characteristics. The underlying motivation for the 
fixed-effects model 1s the following general earnings equation: 

(4) Y 1>t = a + bX 1ft ♦ cP 1ft ♦ e, + e t + e 1ft , 

where as before, Y 1ft 1s the earnings of Individual 1 1n period t; 
X 1ft represent measured characteristics that affect earnings- P< + 

bl i«fr9Lri 0r l !, r0 iH r L. a,n + P K rt1C l pat ! l 0n; and a ' b » and c are Parameters io 
be estimated. Unlike the model described earlier, however, the prrnr 
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serially uncorrelated error term. It 1s also assumed that thes? error 
components are Independent. If the process of selecting Individuals 
Into JTPA 1s unrelated to these unmeasured components, th*n an unbiased 
estimate of c could be obtained by estimating the abo/e earnings 
equation using OLS. If, however, selection 1s related to the 
unmeasured components, then OLS estimates of this equation will yield 
biased program effects. 

The fixed-effects model 1s designed to handle situations 1n which 
selection Into the program Is based on e 1? the 1nd1v1dual-spec1f 1c 
unmeasured characteristics that are constant over time. 52 
Specifically, 1f Individuals either choose to participate 1n JTPA or 
are selected by program operators on the basis of e^, the bias caused 
by the correlation between the error term and participation can be 
eliminated by differencing the Impact equation over two time periods. 
That 1s, for a post-program period s, and a pre-program period k, 
differencing yields the following estimation equation: 

< 5) Y 1,s" \k = (V V * b <*1,s- X 1,k> * c \t + < e 1,s - e 1,k>- 

As can be seen from this equation, by calculating the change 1n 
earnings from a pre-program to a post-program period, the permanent 
component of earnings due to unmeasured characteristics Is eliminated. 
In addition, since most exogenous measured characteristics such as race 
and education are constant over time, the permanent components of 
earnings due to measured characteristics (bX^ will also be 
eliminated, except perhaps for age. Thus, If selection Into JTPA 
depends only on unmeasured characteristics that do not change over 
time, 1f the Cov(e is - e,*^) = 0, and If the earnings 
structures of participants and comparison group members are similar, 
then regressing the difference In earnings on a constant, a dummy 
variable for JTPA participation, and age will yield unbiased estimates 
of JTPA. 53 

It Is Important to note, however, that there Is good reason to believe 
that the Cov^ s - e^^) will not be zero, particularly for 
pre-program periods near enrollment. That Is, since program operators 
are evaluated on their success In meeting certain performance standards 
related to participants' post-program employment outcomes, they may 
"cream" by choosing Individuals who become eligible primarily because 
of bad luck. That 1s, they may choose Individuals with good long-term 
earnings prospects but who have large negative transitory components 1n 
earnings 1n the Immediate pre-program period, I.e., large negative 
values of e^ ^ . Such a selection process would Introduce a 
correlation between e^t-1 and P lf and lead to biased estimates of 
program Impacts 1f the Immediate pre-program period were used 1n the 



In the autoregresslve earnings model, the fixed effects are 1n 
part accounted for through the Inclusion of prior earnings levels as 
regressors. 

53 Differencing reduces the degree of any polynomial by a factor of 
one, so that 1f the Initial model Includes age and age-squared, then 
the difference model will only Include age. 
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H f lt ren " equation. In fact, use of the immediate pre-program period 
as the base period for the fixed-effects estimator would result 1n an 
upwardly biased estimate of the net program Impact on earnings, because 
the transitory component of earnings would subsequently return to 
normal and earnings would rise, even In the absence of JTPA. 

It 1s possible, however, to reduce the effects of creaming on e< u by 
using a fixed-effects estimator with a base period considerably prior 
to program participation (e.g., two or three years) as long as the 
random component of earnings 1s not correlated over time. In fact, 1f 
the model Is correct and the random component of earnings 1s 
uncorrected over time, then net Impact estimates using different base 
periods (e.g., two, three, or four years prior to enrollment) should be 
equal. Unfortunately, evidence presented by Ashenfelter (1986) and 
Dickinson, Johnson, and West (1986) Indicates that estimates of program 
impacts on earnings from fixed-effects models vary widely using 
different base years. This extreme sensitivity of the fixed-effects 
estimator to the choice of the base year 1s likely due to the fact that 
the random component of earnings 1s correlated over time, as 
demonstrated by Lillard and Willis (1978) and others. The correlation 
of the random component of earnings over time brings us to the third 
approach to deriving unbiased net impact estimates. 



Symmetric-Difference Estimators 

In a recent paper, Heckman and Robb (1982) demonstrated that a 
fixed-effects or difference-estimator svnmetrlc ab out the decision 
BS£iod_ will yield consistent program effects even if the random 
component of earnings is correlated over time, as long as the earnings 
process is covariance stationary. 5* That Is, under the assumption of 
stationarlty, the difference 1n earnings from a pre-program to a 
post-program period that are equally far from the participation 
decision period is not affected by the transitory decline 1n earnings 
that leads trainees to enroll in JTPA. Thus, 1f the decision period 1s 
t-1, the symmetric-difference estimator is equivalent to the 
fixed-effects model described above, with k ■ s-(t-l). 

Results from Dickinson, Johnson, and West (1986) indicate, however, 
that the symmetric-difference estimator is sensitive to the choice of 
the decision period. This 1s in part because their results were based 
on a sample of Individuals who enrolled in CETA throughout a particular 
calendar year, and for whom the appropriate decision year likely 
differed depending on whether they enrolled early or late in the year. 
That is, Individuals who enrolled very early in the year likely based 
t0 enroll on the previous year's earnings— Indicating 
that the difference estimator should be synwnetrlc about period t-1-- 
whereas individuals who enrolled late in the year probably based their 
decision primarily on the current year's earnings, which would result 
™ * difference estimator symmetric about period t. As a result, very 
different net Impact estimates were obtained when t and t-1 were used 



. Covariance stationary means that the correlation between a 
variable at time period t and t-k is the same for a given k and all t. 
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as the decision year for all enrollees. 



A major advantage of the state-level model 1s that because quarterly 
earnings data will be available, It will be possible to more easily 
Identify the decision period(s). In general, 1t Is likely that the 
decision to participate will depend primarily on the earnings In the 
three to six months before the quarter of enrollment In JTPA. If, for 
example, 1t 1s earnings In the six months before the quarter of 
enrol lment that determine program participation, then for a group of 
trainees who enrolled In the first quarter of 1986, one could obtain 
unbiased net Impact estimates by regressing the change In earnings from 
a pre- to post-program period symmetric around the midpoint of the 
decision period (I.e., symmetric around October 1, 1985) on a constant 
and a JTPA participation dummy variable. 55 For example, to determine 
the net Impact on annual earnings one could use earnings measured two 
years before and after the decision period. That Is, one could use 
earnings from April 1983 to March 1984 (with a midpoint of October 1, 
1983) as the base period and earnings from April 1987 to March 1988 
(wltli a midpoint of October 1 , 1987) as the outcome measure. This 
Illustrates a major disadvantage of the symmetric-difference approach, 
namely Its extensive longitudinal earnings requirements. That 1s, to 
estimate the net Impact on annual earnings using the base periods and 
outcome periods chosen above, one would need five years of data, and 
the results would not be available In a timely fashion. 

Although 1t Is possible to use other base and outcome periods that are 
symmetric about the decision period, but closer to the decision period, 
so that results could be available In a timely faslon, other problems 
arise. For example, If one used earnings from April 1984 to March 1985 
as the base period (I.e., one year prior), then the appropriate outcome 
period would be from April 1986 to March 1987. This period 1s 
unsatisfactory since a large portion of Individuals who enrolled during 
the first quarter of 1986 would st1 11 be In the program. Treating 
program participants as having completed the program when 1n fact they 
are still enrolled would lead to a biased estimate of program Impacts. 

In addition to the extensive longitudinal data requirements of the 
symmetric-difference estimator, 1t should be noted that this estimator 
may be sensitive to potential seasonality biases if less than a year 1s 
used as the outcome period. For example, provided all trainees who 
enrolled 1n JTPA during the first quarter of 1986 were terminated by 
October 1, 1986, then to estimate the net Impact on quarterly earnings, 
the first post-program quarter for all participants would be from 
October 1, 1986 through December 31, 1986. This outcome period has a 
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55 It 1s also possible to Include a set of demographic variables to 
control for differences 1n earnings trends among Individuals with 
different measured characteristics. For example, one could Include 
age, ethnicity,^ and education as control variables In the 
symmetric-difference model . However, It is Important that no 
pre-program earnings variables or any of the other variables that 
depend on earnings in the pre-program period (e.g. , Food Stamp 
recipient, WIN registrant) be included 1n the model, as that would 
violate the constraints implicit In the model. 
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midpoint of November 15, 1986, or 13.5 months after the midpoint of the 
assumed decision period for these enrollees. Therefore, the 
appropriate base period for the symmetric-difference model would be 
from July 1, 1984 through September 30, 1984 (with a midpoint of August 
15, 1984 or 13.5 months before the midpoint of the decision period), 
^hat 1s, the symmetric-difference model would use earnings in the 
fourth quarter of 1986 minus earnings 1n the third quarter of 1984 as 
the dependent variable. 56 If there are seasonal differences 1n the 
characteristics of JTPA enrollees or 1f the net Impacts of the program 
vary by season, changes 1n earnings over different calendar quarters 
could give misleading estimates of the overall net Impacts of JTPA. 



Summary and Recommended Approach 

In this section we have presented three statistical models for 
estimating the average net Impacts of JTPA on earnings. Each of the 
models rests on certain assumptions (e.g., that the model 1s correctly 
specified, that all variables are perfectly measured, and other 
assumptions concerning the nature of the earnings generating process, 
the structure of the unmeasured characteristics, and the process that 
determines selection Into JTPA) that may be difficult to meet 1n 
practice. In addition, the models differ somewhat 1n their data 
requirements and ease of use. Below, we briefly summarize the 
strengths and weaknesses of the various approaches and present a 
recommended analysis strategy. 

The analysis of covarlance or autoregresslve earnings model controls 
for differences in measured characteristics between participants and 
comparison group members that remain after the match 1s drawn, but does 
not control for any unmeasured differences that affect both earnings 
and program participation that may be Introduced by the various 
selection processes. It can provide consistent estimates of program 
effects 1f program participation depends only on pre-program earnings 
or on other measured characteristics. This estimation model 1s quite 
simple to Implement, has been used extensively 1n the literature, and 
has relatively modest data requirements. 

The other two types of estimation models make assumptions about how 
unmeasured characteristics affect earnings and then propose changes 1n 
earnings models from a pre-program to a post-program period to reduce 
the effects of unmeasured characteristics. The fixed-effects model 
assumes that the effects of unmeasured characteristics on earnings are 
constant over time, whereas the symmetric-difference estimator 1s 
somewhat more general and allows the effects of unmeasured 
characteristics to be correlated over time, provided the earnings 



Other potential dependent variables for measuring the net Impacts 
on quarterly earnings Include earnings 1n the first quarter of 1987 
minus earnings 1n the second quarter of 1984, earnings 1n the second 
quarter of 1987 minus earnings 1n the first quarter of 1984, and so 
on. The point 1s that the pre- and post-program quarters Involved 1n 
the symmetric-difference estimator always correspond to different 
calendar quarters. 
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generating process 1s covarlance stationary. Each of these approaches 
1s somewhat more complex and 1s likely to be less Intuitive to 
stats-level analysts, which would make the results more difficult for 
them to explain and defend. In addition, these models have 
considerably greater data requirements, which will limit their 
usefulness 1n states that do not keep detailed longitudinal UI earnings 
records, and the results will generally n >t be available on as timely a 
basis. 

Based on considerations related to data requirements, ease of 
Implementation, and understanding, the autoregressl ve earnings model 1s 
preferable. Moreover, based on limited evidence, 1t appears that net 
Impact estimates from autoregresslve earnings models are similar to 
these obtained from symmetric difference estimators when the 
appropriate decision year 1s used. 57 As such, we recommend that the 
autoregresslve ear nings model be the primary method for estimating the 
net Impact of JTP A. but that appropriate cautions be made about the 
potential biases. (For more detailed discussion on how to Implement 
this model for estimating JTPA net Impacts, refer to Volume VI 1n this 
series.) In addition, we recommend that states with detailed 
longitudinal UI earnings records (five years or more) consider using 
the symmetric-difference estimator to determine 1f correcting for 
unmeasured characteristics affects the estimated net Impacts. 



OBTAINING NET IMPACT ESTIMATES FOR VARIOUS SUBGROUPS 

The models described above have focused on providing overall estimates 
of the net Impacts of JTPA for adult men and women. It 1s also of 
considerable policy Importance to determine whether the effectiveness 
of JTPA varies by the type of service provided or by Individual 
characteristics. Because these factors may change considerably over 
time, knowledge of how program net Impacts vary among them would help 
Interpret time trends 1n JTPA's Impacts. Furthermore, Information on 
which programs work best for which types of participants and under what 
conditions could provide valuable Information for targeting future 
employment and training programs to the disadvantaged. Although the 
approach to estimating net Impacts for different subgroups 1s formally 
Identical, whether the subgroup refers to program service type or 
Individual characteristics, additional selection biases are likely to 
arise 1n some situations. Below we describe how to modify the 
autoregresslve earnings model described above to estimate the net 
Impacts of JTPA for various subgroups and Indicate the additional 
biases that can arise. 



Results from Dickinson, Johnson, and West (1986) Indicate that 
the autoregresslve earnings model and the symmetric-difference 
estimator yield very similar results for adult men when the appropriate 
decision year 1s used. However, the net Impact estimates for adult 
women are somewhat more positive when the symmetric-difference 
estimator 1s used. This suggests that correcting for differences 1n 
unmeasured characteristics may be more Important for women, and to the 
extent that the selection processes 1n JTPA are similar to those 1n 
CETA, the estimates from an autoregresslve earnings model may be lower 
bound estimates of the net Impacts for adult women. 
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In general, subgroup effects are estimated by including in the 

[2 re „ S ri°it eqU « a f t1 ;; a " " 1nterac t 1 °"" term, that 1s, a term representing 
the product of the dummy variable for JTPA participation with the 
variable for the subgroup of Interest. Suppose one Is Interested 1n 
testing whether the net Impact varies by the characteristic represented 
? Jl 6 .J h [ ee c dummy var1ables z l. *2. and Z 3 . For example, one 
mignt think of the three variables as representing race/ethn1c1tv 
categories (white, black, other), or program services (CT.Sjt! 
J^Il Then using the autoregresslve earnings uodel described 
above, one would estimate the following model to determine whether JTPA 
net Impacts differed across these subgroups: 

(6) Y 1.s ' * * b *l,t-l + VZ 1 * £ Vl.t-J ♦ * liS ♦ c^, ^ 

+C 2 P 1,t Z 2 +c 3 P 1,t Z 3 +e 1,s- 

That 1s, the only change to the basic model described 1n Equation (3) 
Involves replacing the program participation dummy variable with three 
variables that each Involve the JTPA dummy variable multiplied by one 
of the three variables representing the particular subgroup. Then ci 
represents the net Impact for the first subgroup (e.g., whites) ci 
represents the net impact for the recond subgroup (e.g., blacks),' and 
c 3 is the estimate of the net impact for the third subgroup (e.g. 
other race/ethnidty groups). 

To formally test whether the program net impacts differ significantly 
across the subgroups of Interest, an F-test is used. Similar to the 
F-test described earlier 1n this chapter, this test involves a 
comparison of the residual sum of squares from two regression 
equations, the main net Impact equation (1n which the treatment 
variable 1s entered directly and 1s not interacted with any subgroup 
characteristic), and one 1n which the main treatment variable 1s 
omitted and the treatment 1s interacted with variables representing 
each subgroup. Specifically, the following test statistic would be 
computed: 

(7) [RSS m - RSS^/r 

RSS^N-K) 

where RSS m and RSS 1 are the residual sum of squares from the main 
model and the model with subgroup Interactions respectively, r is the 
number of restrictions imposed by the main model (I.e., the number of 

CO 

It should be noted that, 1n principle, similar analyses could be 
performed to determine whether net Impacts vary across local labor 
market conditions. However, because the labor market variables would 
take on the same value for all persons 1n the same local area in a 
given time period, there 1s not likely to be sufficient variation 1n 
these variables to obtain precise estimates of how program impacts vary 
across local labor market conditions, except in large states, with many 
SDAs, and where there are considerable differences in labor market 
conditions across SliAs. 
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subgroups minus one), and N-K 1s the number of degrees of freedom In 
the main Impact model. This test statistic follows an F(r,N-K) 
distribution, and the null hypothesis that the net Impacts do not vary 
across the subgroups of Interest (e.g., across racial groups) would be 
rejected for r=2 and sufficiently large sample sizes at a .05 ( .01) 
significance level If the test statistic exceeded 2.99 (4.60). 

Although the F-test described above Is the appropriate test for the 
equality of the net Impacts across several subgroups, one can also 
provide Information on differences In net Impacts by subgroups using an 
alternative estimation strategy. For example, consider the following 
regression model : 

k 

(8) Y 1#s = a + bX 1>t _! + vZ, + ^ d 3 Y 1,t-j + ^1,5 * c^M + ^1,^2 
+ ^1,^3 + e 1,s- 

In this formulation, the program dummy variable and all except one or 
the Interaction terms are Included In the model ; that Is , Z] becomes 
the "left-out category." Then, ci Is an estimate of the net Impact 
for subgroup Z] as before; n;> Is an estimate of the extent to which 
the program Impact differs for subgroup Zg relative to subgroup Z*| ; 
and n3 measures the extent to which the program Impact for subgroup 
Z3 differs from that for subgroup Z] . In fact, the two equations 
are essential ly equivalent In that c 2 = C] + n 2 , and 
C3 = ci + n3. The advantage of estimating the equation 1n the 
latter form Is that one can directly test which program Impacts are 
significantly different from the left-out category, because the 
standard errors of n 2 and n3 are available. That 1s , separate 
t-tests of the estimated coefficients n 2 and n3 provide Information 
on whether the estimated Impacts for the two subgroups (Z 2 and Z3) 
are significantly different from the Impact for Zi . On the other 
hand, 1f one wants to determine whether each of the subgroup Impacts Is 
significantly different from zero, then estimates of the standard 
errors of c-| , c 2 , and c 3 are requl red , and the earl 1er 
specification should be used. 

It should be noted that In attempting to disaggregate JTPA net Impacts 
across subgroups, 1t 1s Important that the subgroup characteristics 
also be 1ncl uded 1n the model as control variables to account for 
differences 1n the general level of earnings across these subgroups. 
That 1s, in our Illustration, the three Z^ variables must also be 1n 
the model separately so that the estimated coefficients (I.e., the 
c^'s or n^'s) only capture earnings differences due to JTPA across 
these subgroups and do not Include the average earnings differences due 
to the Z^'s themselves. In addition, In order to ensure that the 
c^'s are capturing meaningful differences 1n program net Impact by 
subgroups, It 1s Important that the subgroups be mutually exclusive and 
exhaustive. That 1s, 1f the equation above that has the program 
participation dummy Interacted with all the subgroup dummy variables of 
Interest had omitted one of the Interaction terms, then those 
observations would be 1n fact treated as part of the comparison group, 
and biased estimates of the net Impacts of JTPA for different subgroups 
would be obtained. Fortunately, this 1s a simple problem to avoid. 




The Interaction analyses described above will Identify the types of 
Individuals who benefit roost from JTPA and whether there Is a general 
pattern to the variation 1n program effectiveness. Because 
Individuals 1 pre-prograrn characteristics cannot be affected by JTPA, no 
additional selectivity bias 1s Introduced In disaggregating JTFA net 
Impacts by demographic subgroups. 59 However, as we describe below, 
this may not be the case when examining whether JTPA effectiveness 
varies by program activity or by other aspects of the treatment 
provided. 

In principle, to probe beneath the average not Impacts of JTPA and 
provide Information on the program activities and other aspects of the 
treatment (e.g. , length of stay) that contributed to the average 
effects, one would perform an Identical Interaction analysis to the one 
described above, except that the subgroup characteristics (I.e., the 
Zj's) would relate to JTPA services received. For example, to 
examine net Impacts by program activity, then If , Z2, and Z3 
corresponded to classroom training, on-the-job- training, and Job 
search assistance, respectively, then c«| would be the estimate of the 
average net Impact for CT, cp would represent the estimated net 
Impact for OJT, and C3 would represent the estimated net Impact for 
JSA. However, as we describe below, there Is a major problem that 
threatens the Internal validity of the by-program activity net Impact 
analysis. 

The problem Is the familiar one of selection bias. In this context, it 
relates to the nonrandom assignment of program services to JTPA 
participants. As described above, the assignment of program activity 
1s likely to be based on the agency's perception of an individual's 
needs and abilities. To the extent that this assignment process 1s 
based solely on the measured characteristics of participants (i.e., 
age, race, sex, education, pre-program earnings), this will not bias 
the net Impacts by program activity as these characteristics will be 
included in the net Impact model. A much more serious problem arises 
1f the assignment of program activities Is based on unmeasured 
characteristics, such as motivation and ability, and those unmeasured 
characteristics also affect earnings. If, for example, the 
more-motivated participants are assigned to OJT programs and are also 



D * If the characteristics defining the subgroups of Interest are not 
measured equally well for the participants and comparison group 
members, however, the subgroup Impacts will Inappropriately reflect 
these differences. That 1s, because the presence of measurement error 
In an Independent variable biases Its estimated coefficient downward 
(see discussion 1n later section), 1f the amount of measurement error 
on a subgroup characteristic were greater in the JTPA sample, for 
example, than 1n the comparison group, the effect of that 
characteristic on earnings would be smaller 1n the JTPA sample than 1n 
the ES registrant sample. The Interaction term would Inappropriately 
pick up such a difference and mlsleadlngly Indicate that JTPA Impacts 
were smaller for Individuals with that characteristic. It may be 
useful, therefore, whon examining the quality of the comparison group 
to make sure that the variables used 1n the Interaction analysis do not 
have preexisting differential Impacts 1n the two samples. 
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more likely to have higher earnings, then the estimated coefficient of 
OJT (c 2 ) may be large and positive. However, this significant 
coefficient of OJT on earnings would not Indicate a causal 
relationship, but merely reflect the fact that more-motivated 
Individuals were assigned to the OJT activity. Thus, one must be very 
careful In Interpreting net Impacts by program activity because of this 
additional selection bias that can occur as a result of the assignment 
process. 

As described 1n the literature review (see Chapter 2), there are 
statistical procedures that can potentially be used to correct for such 
selection biases. These procedures rely on Instrumental variable 
techniques to produce consistent estimates of program Impacts In the 
presence of selection bias . However, as noted above, such procedures 
are only useful If one can Identify variables to play the role of 
Instruments. That Is, one must find variables that affect the 
assignment of program activities to particular Individuals but that do 
not affect participants' earnings. Although this 1s a difficult task 
1n any circumstance, It will be particularly difficult given the 
limited number of Individual characteristics available for the proposed 
analysis. 

It may, however, be possible to Identify potential Instrumental 
variables from a carefully structured process analysis. That 1s, an 
Important aspect of the process analysis will be a detailed description 
of the process Involved In assigning program activities to specific 
participants. Such an analysis might Indicate, for example, that the 
assignment of program activities Is Influenced primarily by the 
availability of program-activity slots In a given SDA, and since the 
availability of program slots should not affect post-program earnings, 
such a variable could potentially serve as an Instrument. 

Even 1f the process analysis Is not successful 1n Identifying specific 
variables to play the role of Instruments, It could shed light on the 
relative validity of the program-activity net Impact estimates and 
provide some Indication of the likely direction of the selection 
biases. For example, the process analysis should be able to determine 
to which program activities the more- and less-job ready participants 
are assigned. 60 Then, If the Impact analysis finds that a given 
program activity has a large effect on post-program earnings and the 
process analysis Indicates that the more disadvantaged Individuals are 
assigned to that activity, It would Increase confidence that the 
observed relationship 1s causal and not due to selection bias. If, 
however, the process analysis Indicates that the Individuals assigned 
to that program activity are considerably more advantaged, then one 
should not be very confident that the net Impact estimates for that 
program activity solely reflect the effects of the program. 

In addition to providing evidence on how the net Impact of JTPA varies 
by Individual characteristics and program activities, 1t may be useful 



6 ^ A comparison of the differences 1n measured characteristics of 
participants across program assignments should al so provide some 
Information on the probable direction of the selection bias. 
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to examine whether the Impact of JTPA varies by participants 1 program 
experiences. For example, several studies have attempted to determine 
how the Impacts of employment and training programs vary with length of 
stay In the program. Although the results of such analyses could 
provide Important Information about the mechanisms through which 
employment and training programs produce their effects, as we describe 
below there are Important selection bias problems that limit the 
validity of such analyses. 

To Investigate whether the net impacts of JTPA vary by program length 
of stay, one would estimate an autoregresslve earnings model with the 
overall program participation dummy variable Interacted with variables 
representing length of participation. For example, using the notation 
developed above, one could categorize the length of stay variable Into 
four dummy variables corresponding to stays of less than one month, one 
to three months, three to six months, and greater than six months. 
Then the estimates of the four c^ coefficients would represent the 
average Impacts of JTPA for Individuals with these different lengths of 
stay. Alternatively, If the effects of length of stay on earnings are 
approximately linear, one could estimate a model with a JTPA 
participation dummy and the participation dummy Interacted with actual 
weeks In the program less average number of weeks in the program, and 
obtain a more efficient estimate of program effects. In this 
formulation, the coefficient of the JTPA dummy represents the estimated 
Impact of JTPA at the average length of stay, and the coefficient of 
the Interaction term Is an estimate of the dollar Impact of an 
additional week of program participation. 

Although 1t Is straightforward to construct such program experience 
variables and estimate the coefficients of the Interaction terms 1n the 
net Impact models, one must be very careful 1n Interpreting the 
results. Once again, this Is due to the familiar problem of selection 
bias. Although the autoregresslve earnings model controls for 
differences on measured characteristics between short- and long-term 
participants, 1t Is likely that some differences on unmeasured 
characteristics remain. Individuals who leave the program early may be 
less motivated or, alternatively, may have found employment on their 
own. On the other hand, Individuals who stay 1n the program a long 
time may do so because they have fewer other employment opportunities. 
Length of stay 1s also likely to depend on the type of program activity 
and SDA characteristics. 

Because of these additional selection bias problems, one should use 
extreme caution 1n Interpreting the estimated Impacts by length of stay 
as representing causal relationships. In order to overcome these 
biases and obtain estimates of the net Impacts of JTPA by program 
length of stay that one has confidence In, It 1s necessary to formally 
account for the endogenelty of the length of stay variables, which 
would require Instrumental variable procedures. Operationally, one 
would first estimate a regression equation to obtain a predicted value 
of length of stay. Such equations would be estimated using the 
participant sample only, and the predicted values would be entered 1n 
the net Impact equation for participants, and zeros would be Included 
for comparison group members. 
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The success of such an Instrumental variable procedure relies heavily 
on the ability to Identify variables to play the role of Instruments. 
That Is, one must find variables that are highly correlated with length 
of stay but that do not directly Influence earnings. Depending on the 
types of Information obtained from participants and Included In the 
JTPA HIS, one may be able to obtain variables that can play the role of 
Instruments for length of stay. For example, It may be possible to 
develop Instruments from answers to very simple questions about 
participants 1 general satisfaction with JTPA services and whether 
participants were assigned to the type of program they wanted. That 
1s f 1t seems plausible to assume that Individuals who are satisfied 
with the program, or who are assigned to the program type they were 
most Interested In, would remain In JTPA longer. At the same time, 
however, there Is no obvious reason why satisfaction with the program 
or assignment to the desired training program should affect earnings, 
Independent of the effect of the training and the actual length of 
stay. Thus, these variables could be entered Into a regression 
equation along with other demographic characteristics to explain 
program length of stay, and then a predicted value for length of stay 
could be constructed and entered In the net Impact equation 1n place of 
the actual length of stay. 

To summarize, estimating how the net Impacts of JTPA vary by program 
experiences Introduces an additional selection bias that Is In practice 
difficult to overcome. For the most part, these additional biases have 
been recognized In the literature, but not explicitly dealt with due to 
the lick of variables to play the role of Instruments. In the absence 
of adequate Instruments, one must not place too much confidence- 1n the 
estimated Impacts. For states that are very Interested In overcoming 
these selection bias problems, they should carefully review what 1s 1n 
the JTPA HIS, as well as the Information obtained during the process 
analysis, to see If variables that affect length of stay, but not 
earnings, can be identified. If successful, they should Implement the 
Instrumental variable approach and determine whether the variables 
selected 1n fact strongly affect length of stay. If not, then 1t 1s 
not necessary to go to the second stage of Including the predicted 
value 1n the net Impact model, as the predicted variable will be too 
highly correlated with other characteristics 1n the model , and the 
results will not be reliable. If, however, the Instruments do strongly 
affect length of stay, then the net Impact model should be estimated 
with the predicted length of participation replacing the actual value. 
If the Instruments have been successful, then they will purge the 
correlation between the error term and length of stay and result 1n 
consistent net Impact estimates. 



MEASUREMENT ERROR AND OTHER STATISTICAL ISSUES 

In estimating the net Impacts of JTPA programs on earnings and AFDC 
grants, we have recommended that ordinary least squares (OLS) 
regression techniques be used. Although such techniques are easy to 
Implement and are generally very robust, 1t 1s Important to recognize 
that, depending on the specific outcome measure and the population 
group of Interest, OLS regression may not be the most appropriate net 
Impact estimation technique. At the same time, however, the 
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alternative models that may be more appropriate are 1n general not 
contained In standard statistical software packages and thus will not 
be readily available to state-level researchers. Below we briefly 
discuss some of the limitations of OLS regression models for the 
planned analyses and briefly Indicate other statistical models that 
could be used. 



Measurement Error 

In the regression models described above, we have treated the variables 
as though they are perfectly measured, that 1s f as though they contain 
no measurement errors. Although this Is a very convenient assumption, 
1t 1s also somewhat unrealistic, in fact, It Is likely that most 
variables contain some measurement error, although the nature and 
extent of the error may be such as to not affect the estimated program 
net Impacts. Below we discuss the Implications of various types of 
measurement error for the state-level net Impact tfodel. 61 

We first consider the Implications of measurement errors In the 
dependent variable. As described In Chapter 3, In developing 
post-program earnings variables from State UI Wage Records, these 
variables will be subject to measurement errors for Individuals who 
work 1n nonreported employment or who work 1n another state. In both 
cases, zero earnings will be recorded 1n the data set when the true 
value 1s some positive amount. Such measurement error In the dependent 
variable can bias the estimated net Impacts of JTPA on earnings If 
treatment status Is correlated with the measurement error 1n earnings. 
That 1s, If the likelihood of working In nonreported (or out of state) 
employment Is different for JTPA participants than for comparison group 
members, OLS estimates of program net Impacts will be biased. 
Specifically, 1f JTPA participants are more (les5* likely to work 1n 
nonreported employment or out of state, the estimated Impacts will be 
biased downward (upward). If, however, the measurement error In the 
dependent variable Is uncorrelated with treatment status, the estimated 
program net Impact will not be biased* 

It should be noted, however, that even though the net Impact estimates 
may not be affected by measurement error 1n the dependent variable, the 
precision of the estimated program effects will generally be affected. 
Assuming that the measurement error 1n earnings and the error term 1n 
the net Impact model are uncorrelated (or positively correlated), the 
presence of measurement error Increases the variance of the estimated 
net Impacts. Thus, 1f one suspects that the dependent variable 1s 
likely to suffer from considerable measurement error, 1t Is possible to 
compensate for the expected loss of precision by drawing larger 
analysis samples. 

We next consider the complications that are Introduced when there 1s 
measurement error 1n an Independent variable. Independent variables 
may be measured with error because of recording or keypunch errors, 



For additional Information see Duncan (1975), Goldberger and 
Duncan (1973), or Rao and Hlller (1971). 
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because of data editing procedures that assign default values when the 
data Item 1s missing for a person, or as a result of Inappropriate 
aggregation. For example, measurement error can be Introduced by 
collapsing heterogeneous JTPA treatments Into one common treatment 
dummy variable. That 1s t by representing the effect of JTPA by a 
simple participant dummy variable that combines diverse treatments such 
as classroom training, 0JT t or Job Search Assistance that have 
different lengths of participation and Intensity, one necessarily 
Introduces measurement error Into the treatment variable. As 1s well 
known (see, for example, Duncan, 1975), measurement error biases the 
coefficient of the Independent variable toward zero. This Is true for 
alj. independent variables with measurement error, that is, variables 
representing the treatments provided by JTPA as well as variables 
Included to control for differences In measured characteristics between 
participants and comparison group members. The magnitude of the bias 
depends on the size of the variance In the measurement errors relative 
to the variance of the error term In the net Impact model. 
Specifically, the greater the variance In the error of measurement-- 
that 1s, the lower the accuracy of measurement—the greater Is the bias 
towards zero. This emphasizes the Importance of developing Independent 
variables that accurately measure the extent of the treatment provided, 
which 1n turn reinforces the Importance of conducting a process 
analysis to Identify the major differences and similarities 1n program 
treatments. 

As the above discussion Indicates, measurement error Is an Important 
conceptual Issue that analysts should be aware of when specifying their 
models and Interpreting the results. In addition, 1t should be noted 
that approaches have been developed to formally Incorporate measurement 
error Into standard statistical models (see, for example, Joreskog, 
1973). However, these multiple Indicator approaches are very complex 
and generally beyond the scope of the state-level net Impact model. As 
a result, states that anticipate severe measurement problems should 
consult a measurement specialist, perhaps from within their university 
system. 



Dichctoraous Dependent Variables 

The models described earlier In this chapter have concerned continuous 
outcome measures such as earnings and AFDC grants. However, as 
described 1n Chapter 3, some of the outcome variables to be examined 
are dlchotomous. Examples Include whether an Individual 1s employed or 
whether he or she 1s receiving AFDC payments during a particular 
period. In the case of dlchotomous outcome variables, the prediction 
from a regression model represents the probability of the Individual 
being employed or on welfare. However, a standard OLS regression model 
suffers from two deficiencies 1n this circumstance. First, the 
predicted probability 1s not constrained to fall between 0 and 1 , and 
thus nonsensical results can potentially arise. Second, the error term 
1n the model 1s necessarily heteroscedastlc , and OLS estimation 1s 
Inefficient. 

A number of statistical procedures have been designed to deal with 
these circumstances, the most popular being loglt and problt 
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analyses 62 Although such models are theoretically more appropriate 
for dlchotomous variables, 1n practice the predicted probabilities are 
likely to be quite similar to OLS estimates, provided the dlchotomous 
outcome measure does not represent an extremely rare or extremely 
likely event. That 1s, as long as the mean outcome measure 1s 
approximately between .2 and .8, OLS estimates of a linear probability 
mode usually yield very similar predictions to loglt or problt 
models. Because this will generally be the case for the outcome 
measures to be examined 1n the net Impact model, the deficiencies of 
OLS techniques should not be a serious problem for dlchotomous outcome 
variables 1n the state-level model. 



Bounded Dependent Variables 

A related statistical issue arises for outcome measures that although 
continuous, have an upper or lower bound at which a large number of 
observations occur. for example, Af DC grants are zero for 
nonpartldpants and earnings are zero for nonworkers. In the 
relatively short post-program period, many Individuals can be expected 
to have zero earnings, and even more Individuals are likely to have 
zero Af DC grants. If there are numerous observations at the limit 
ordinary least squares will be Inconsistent and will also be 
Inefficient, because the error term 1s heteroscedastlc . One way of 
accounting for the statistical problems 1s to estimate a toblt model 
(Tobln, 1958), which 1s designed to handle outcome variables that are 
truncated normal, I.e., normally distributed but truncated at a 
particular point. Unfortunately, there 1s extensive evidence that the 
distributions of earnings for adult men and women are not truncated 
normal 1n which case the toblt specification 1s Inappropriate and could 
yield misleading results. 

Thus, although 1t 1s Important to recognize the limitations of OLS 
techniques, the potential deficiencies of alternative procedures, 1n 
combination with their much greater expense and limited availability 
suggest that the state-level model should rely on OLS techniques 



ADJUSTMENTS FOR POTENTIAL DATA AND DESIGN DEFICIENCIES 

The models described above Involve a comparison of the UI earnings 
records of JTPA participants with those of a comparison group derived 
from ES registrants. In addition to the problem of selection bias, 
there are some deficiencies In the UI Wage Records and In the ES data 
that may affect the results. For example, as Indicated above, UI Wage 
Records contain measurement error 1n that they do not reflect earnings 
from jobs that are not 1n reported employment or earnings from jobs 
that are located across the border In other states. The ES data are 
deficient because there Is Inadequate Information on whether ES 
registrants participated 1r. JTPA, which may result In a "contaminated- 
comparison group. In this section, we briefly discuss the likely 
extent to which the basic Impact estimates will be affected by these 



for a discussion of these alternative models see Thell (1971). 
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data and design deficiencies and Indicate the types of adjustments that 
may be necessary. 

To the extent that 1t 1s Impossible to Identify individuals In the 
comparison group who participated 1n JTPA, the comparison group will be 
"contaminated." Such contamination would lead to an underestimate of 
the net Impacts of JTPA, since 1t would effectively dilute the 
treatment as comparison group members would have also received JTPA 
services. We do not believe, however, that the net Impact estimates 
will be seriously affected by the potential contamination problem. 
This Is so for two reasons. 

First, it may be possible to reliably Identify ES registrants who have 
received JTPA services. This could be done either by comparing the 
social security numbers of comparison group members with current and 
past lists of JTPA participants and excluding all matches. 
Alternatively, the ES service file Includes Information that allows one 
to Identify Individuals who recently enrolled In JTPA programs. 
Although this Information Is likely to be Inadequate both because of 
underreporting problems and the fact that the information only relates 
to the current year, It can be used to minimize the contamination 
problem. 

Second, we believe that the magnitude of the contamination problem is 
likely to be small. Although the ES is one source of applicants for 
JTPA programs and, as such, one might expect that contamination could 
be high, existing data suggest otherwise. For example, based on data 
for the State of Washington for program year 1985, only 0.1 percent of 
all ES registrants active during the year enrolled in JTPA programs. 
Moreover, only 0.3 percent of those economically disadvantaged enrolled 
*n JTPA. Although the figures are somewhat higher for enrollment 1n 
any training program (e.g., JTPA, Job Corps, WIN, other)— 1.0 percent 
for all applicants and 3.1 percent for those economically disadvantaged 
--even these participation rates are small enough to be safely Ignored. 

In states that for some reason have considerably higher probabilities 
of ES registrants enrolling In JTPA, it may be necessary to make some 
adjustment to the net Impact estimates. Although Individual 
adjustments are not possible, one can make an aggregate adjustment. 
Specifically, if p Is an estimate of the proportion of the comparison 
group participating in JTPA, the estimated program net impact should be 
multiplied by 1/(1 -p) to obtain the true impact. 63 

The major limitation to the UI Wage Records 1s that some jobs are not 
in reported employment and earnings from those jobs are therefore not 
included 1n the post-program earnings measure. As discussed above, the 
omission of nonreported earnings (i.e., measurement error) can bias the 
estimated Impacts of JTPA if program participation affects the 
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63 Because the autoregressl ve model controls for the Impacts of 
previous participation, an adjustment should be made for each period 
from the decision to enroll through the post-program period of 
Interest. Thus, p 1s properly thought of as a cumulative participation 
rate that 1n this application spans roughly a one to two year period, 
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probability of working 1n nonreported employment. Although this was 
an Important concern of earlier studies, particularly given the high 
likelihood of CETA participants turning their training slots into 
subsequent jobs 1n the public sector (which are less likely to be 1n 
reported employment), given the focus of JTPA on employment 1n the 
private sector, this should be less of a problem for the state-level 
model . 

There 1s very little Information available on the extent to which 
nonreported earnings 1s likely to bias net Impact estimates. The only 
study that has provided any evidence on the Impact of employment and 
training programs on the likelihood of working in nonreported 
employment 1s Dickinson, Johnson, and West (1986). Using SSA earnings 
records and Interview-reported earnings, they created a measure of 
whether a person was working 1n nonreported employment (I.e., SSA 
earnings of zero and positive Interview-reported earnings) and examined 
whether CETA participants were more likely to be working 1n uncovered 
employment In the post-program period. They found that adult male 
participants were generally slightly less likely to be working in Jobs 
that were not covered by Social Security, but that there were no 
differences for adult women. Thus, if these results also applied to UI 
earnings reporting problems and to the sample of JTPA participants and 
comparison group members selected for the state-level model, this would 
suggest that the underreporting problem could generate net Impact 
estimates for men that are somewhat overstated, but that no adjustments 
for women would be necessary. However, since these results do not 
directly apply to the Issues facing the state-level model, one should 
be reluctant to use such results to adjust net Impact estimates for the 
reporting problems of UI Wage Records. 

In the absence of additional Information, 1t will be very difficult to 
determine the extent to which JTPA Impacts are affected by the 
Incomplete coverage of UI Wage Records. To determine whether 
participants and comparison group members differ 1n terms of their 
likelihood of working 1n nonreported employment, one would need 
Information on Interview-reported earnings and UI Wage Records for both 
groups In the post-program period. Alternatively, 1f one had 
Information on the Industry of employment of all Individuals, one might 
get some sense of whether the earnings patterns are consistent with the 
nature of nonreported employment across Industries. However, because 
such d£ia must come from surveys of both participants and comparison 
group members, 1t 1s unlikely that such Information will be available 
for the net impact model. In that event, the best one can do 1s 
acknowledge the potential problem and Indicate that the net Impact 
estimates assume that JTPA does not affect the probability of working 
1n nonreported employment or Implement more complex methodologies that 
directly Incorporate measurement error. 

Finally, the most Important potential adjustment to be considered 1s 
for selection bias, that 1s, systematic differences between 
participants and comparison group members that cannot be directly 
controlled for 1n the autoregresslve net Impact model. One can attempt 
to adjust for selection bias by using evidence on pre-program 
differences between the two groups. For example, one could use 
estimates of differences 1n adjusted pre-program earnings or AFDC 
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grants between participants and comparison group members that are due 
to unmeasured characteristics that are developed as part oF the 
analysis to examine the adequacy of the comparison groups described 
earlier In this chapter. The size of the estimated difference In 
pre-program earnings due to unmeasured characteristics 1s a reasonable 
estimate of the amount by which earnings Impacts could be overstated or 
understated {depending on whether 1t 1s positive or negative) If the 
difference persisted In the post-program period. Thus, for example, If 
adult men (women) JTPA participants are estimated to have earned $100 
more ($200 less) In the Immediate pre-program year than Individuals 1n 
the comparison groups, using this first approach one would adjust the 
main Impact estimate (I.e., the coefficient of the JTPA dummy variable) 
downward (upward) by $100 ($200) for men (women). It should be noted, 
however, that because pre-program earnings are controlled for In the 
autoregresslve net Impact model, this adjustment may overcompensate for 
the selection bias due to differences In unmeasured characteristics 
between the two groups. 
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EXHIBIT A-l 
Washington State JTPA Application Form 



APPLICATION FORM 
Job Training Partnership Ac! Program 

SUB CODE 

SSN »!»' - I I I . I 1 I I I . _ 




Phone (l_L_LJ) L_LJ_j . LJ_LdU Message Phone ( l I I | ) III! . Li 



Date of Birth 

LU/LU/UU 
M M D D Y Y 

Race/Ethnic Group I I 

1 White 

2 Black 

3 Hispanic 



A <>« Sex U 

F Female 
M Male 



4 American Indian/ 
Alaskan Native 

5 Asian /Pacific 
Islander 
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SSN ' ' I I • U_J - Mill - U Name 



Application Date l_LJ/LJLJ/LJ_l 



SELECTIVE SERVICE I 1 

1 Registered 

2 Not registered 

3 Not applicable 



CITIZENSHIP I I 

1 US Citizen 

2 Eligible Ncn-Citizen 

3 Non-citizen 



IF ELIGIBLE NON-CITIZEN, 
ALIEN CARD NUMBER 



FOSTER CHILD I I HANDICAPPED I I SSI I 1 

1 Yes 1 Physical 1 Yes 

2 No 2 Mental 2 No 

3 Not Applicable 



RECEIVING PUBLIC ASSISTANCE 

AFDC I I CASE NO. LJ-J_l— I— l— l— l— LJ - l_l GRANT AMOUNT $ I I I I I I 

1 Yes START DATE / / 

2 No 



REFUGEE 
ASSISTANCE 

1 Yes 

2 No 



I I CASE NO. I I I I I I I I I l - l_i GRANT AMOUNT $ I I I I 



START DATE 



GENERAL 
ASSISTANCE 

1 Yes 

2 No 



I I CASE NO. I I I I I I I I I I - U 

START DATE / / 



GRANT AMOUNT $ 
TYPE 



FOOD STAMPS 

(ONLY) I I CASE NO. I I 1 I I I 1 I I I - U 

1 Yes START DATE / / 

2 No 



WIN REGISTRANT I I UNEMPLOYMENT WEEKLY BENEFIT 

1 Yes COMPENSATION STATUS I I AMOUNT $ I l I 

2 No 1 Eligible Claimant 

2 U.C. Exhaustee 

3 Not Applicable 



OFFICE USE ONLY 
Family Size I I I 

Economically Disadvantaged 

1 Yes 

2 No_ ...... 



Annualized Family Income $ 



Reason 



10% Window Reason 



ERLC 
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(2) 



SSN III! - l_U - I I I I I - LJ NAME 



EDUCATION STATUS I I LIMITED 

1 School Dropout ENGLISH 

2 Student (high school or less) 1 Yes 

3 High School or Equivalent 2 No 

4 Post High School Attendee 



DISPLACED 
HOMEMAKER 

1 Yes 

2 No 



DISPLACED 
WORKER 

1 Yes 

2 No 



MEMBER OF A SEASONAL/ 
MIGRANT FARM FAMILY 

1 Yes 

2 No 



SINGLE PARENT I I TEENAGE 

1 Dependent Children PARENT 

Under Six Years 1 Yes 

2 Dependent Children 2 No 

Over Six Years 

3 Not Applicable 



OFFENDER 

1 Yes 

2 No 



ALCOHOLIC/ 
ADDICT 

1 Yes 

2 No 



VETERAN 

1 Yes 

2 No 



RECENTLY 
SEPARATED 

1 Yes 

2 No 



VIETNAM ERA 
VETERAN 

1 Yes 

2 No 



DISABLED 
VETERAN 

1 Yes 

2 No 



DATES OF MILITARY SERVICE 

Entry / / 

Exit / / 



PRIOR JTPA 
PARTICIPATION 

1 Yes 

2 No 



LABOR STATUS 

1 Employed 

2 Unemployed 

3 Not in Labor Force 



LAST JOB TITLE 

I I I I I I I 



DOT 



LAST 

HRLY WAGE $ 



HOURS 
PER WEEK 



RECEIVED 
LAYOFF NOTICE 

1 Plant Closure 

2 Job Eliminated 

3 Other 

4 No Notice 



DATE NOTICE 
WAS ISSUED 

M M D D Y Y 



OPPORTUNITY FOR 
RE-EMPLOYMENT 

1 Good 3 Poor 

2 Fair 4 N/A 



NUMBER WEEKS 
UNEMPLOYED IN 
LAST 26 WEEKS 

I I I 



NUMBER WEEKS 
EMPLOYED IN 
LAST 13 WEEKS 



CERTIFICATION 

I certify that the information provided is true to the best of my knowledge. I am also aware that the information I have 
provided is subject to review and verification and I may have to provide document to support this application. I am 
also aware that I am subject to immediate termination if I am found ineligible after enrollment and may be prosecuted 
for fraud and /or perjury if I intentionally supplied inaccurate or misleading information. I allow release of this 
information for verification purposes and understand that it will be used to determine eligibility. I have been advised of 
equal opportunity and appeal rights and the Privacy Act of 1974. 

Application 

Date / / 

Signature of Applicant Month Day Vaar 

. * Date 

Signature of Perant, Guardian or Reeponattte Aautt Month Day Yoar 



Date 

Signature of Interviewer Month Day Vaar 



Sub Code I I I I I I 



Counselor I LJ I 
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EXHIBIT A-2 Washington State Employment Security Application For Service 

PLEASE PRINT WORK HISTORY (Minimum last 3 years) PRESS HARD 



MOST RECENT OR PRESENT EMPLOYER 


JOB SUMMARY (doe* what, using wrist, to what) 


ft": 


ADDRESS 


CITY 


STATE 


ZIP 




JOB LOCATION it different 


JOB TITLE 


SALARY 






DATE STARTED 


DATE LEFT 


MONTHS ON JOB 
1 


PAY UNIT 

t—HOURS 3_WEEK 5 YEAR 

2 DAY 4 MONTH 6 OTHER 






REASON FOR 1 VOLUNTARY QUIT 3 LABOR DISPUTE 5 LACK OF WORK 

SEPARATION 9 DISCHARGE (LI NESS ft STH 1 FMPI OYFD 




EMPLOYER 


JOB SUMMARY (does what, using what, to what) 




ADDRESS 


CITY 


STATE 


ZIP 




% 


JOB LOCATION i 


f different 




JOB TITLE 


SALARY 






DATE STARTED 


DATE LEFT 


MONTHS ON JOB 
1 


PAY UNTT 

1_HO'JRS 3_WEEK 5 YEAR 

2 DAY 4 MONTH 6 OTHER 




REASON FOR 1 VOLUNTARY OUrT 

SEPARATION ? DISCHARGF 


fl 1 AROR fMRPIITF ft | / 


KCK OF WORK 




4 ILLNESS 


EMPLOYER 


JOB SUMMARY (does what, uaing what, to what) 




ADDRESS 


CITY 


STATE 


ZIP 






JOB LOCATION if different 


JOB TITLE 


SALARY 




m 


DATE STARTED 


DATE LEFT 


MONTHS ON JOB 
1 


PAY UNIT 

1_HOURS 3— WEEK 5 YEAR 

2 DAY 4 MONTH 6 OTHER 




REASON FOR 1 VOLUNTARY QUIT 3 LABOR DISPUTE 5 LACK OF WORK 

SEPARATION 9 nisrwARfa: s.._ ILLNESS 





3. I HEREBY REGISTER FOR WORK AND/OR REQUEST AN INITIAL DETERMINATION OF BENEFITS POTENTIALLY PAYABLE TO ME UNDER THE WASHINGTON EMPLOYMENT SECURITY ACT 
AND/OR THE FEDERAL UNEMPLOYMENT COMPENSATION ACT. THE BASE YEAR ANO POTENTIAL BENEFITS HAVE BEEN EXPLAINED TO ME AND I CHOSE TO FILE ON THIS DATE. 
I HEREBY CERTIFY THE INFORMATION I HAVE °ROVlDED ON THIS FORM IS ACCURATE. 



FULL SIGNATURE OF APPLICANT 



EMS 5327/511 (1-86) .436- 
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COPY AVAILABLE 



